---
title: "GenAI Dashboard Workshop"
subtitle: "Npuls Lak25 Team"
output: 
  
  flexdashboard::flex_dashboard:
    orientation: columns
    self_contained: true
    social: menu
    source: embed
    code_download: true
---

```{r setup, include=FALSE}
library(flexdashboard)
```

Introduction {data-navmenu=Start}
=====================================

<style>
summary::-webkit-details-marker {
  display: none
}
summary:before {
  background: lightblue;
  border-radius: 1px;
  content: "+";
  color: #fff;
  float: left;
  font-size: 1em;
  font-weight: bold;
  margin: -1px 1px 0 0;
  padding: 0;
  text-align: center;
  width: 12px;
}

.comment {
  background-color: #FFEEAA;
  border: 3px solid #444;
  padding: 1rem;
  margin: 2rem 0;
}

.comment::before {
  content: " INFO ";
  background: #ADD8E6;
  width: 10rem;
  border-right: 3px solid #444;
  border-bottom: 3px solid #444;
  display: block;
  text-align: center;
  position: relative;
  left: -1rem;
  top: -1rem;
  padding: 2px 10px;
  font-weight: bold;
}

</style>

#### Introduction

<p class="comment"><b>This exercise provides intuition on what AI can and cannot do in the context of Educational dashboards. We hope to leave you some breadcrumbs for further exploration.</b></p>

#### Exercise

> Add your best practices in this form at any time.

![Best Practices Input](./IMG/QR_best_practices.png)
* https://forms.gle/uMjqNPT5mhwPmoeu6

#### NOTE

There are many ways to interact with AI. We have chosen one approach for the demo that requires near-zero setting up. However, in a hackathon or development setting, a more permanent environment, we would use tools such as Git for version control and deployment options into the Cloud, such as [Google Collab](https://colab.research.google.com).

***Regards: Alan, Manuel, Priyanka***





Purpose & Limitations {data-navmenu=Start}
=====================================

### Purpose

1. Exploration of Gen AI themes.

### Limitations

1. Lack of dashboard environment
1. No access to the data
1. Non realistic development environment


Collecting Evidence {data-navmenu='Best Practices'}
=====================================

### Evidence for Best Practices

<p class="comment">
The following menu items provide evidence to stimulate the group discussion.
<br><br>
We provide a form to fill in details to support the collection of your ideas on best practices associated with GenAI and Dashboards.</p>


LLM response to best practices {data-navmenu='Best Practices'}
=====================================

### DeepSeek R1 thinks ...

1. ***Clarity and Simplicity:*** \
 Ensure the interface is uncluttered, focusing on essential metrics. Use AI to automatically highlight critical data points, reducing cognitive load.
2. ***User-Centric Design:*** \
 Engage end-users in the design process to understand their needs. Use AI-driven surveys or feedback tools to tailor the interface to user preferences.
3. ***Effective Visualization:*** \
 Utilize AI algorithms to generate intuitive visualizations like heatmaps or trend lines. This helps users quickly grasp data trends without confusion.
4. ***Customizable Views:*** \
 Implement AI-powered customization options that adapt based on individual teaching strategies, offering dynamic and relevant insights.
5. ***Accessibility Features:*** \
 Integrate AI-driven features like voice commands and screen readers to ensure the dashboard is usable by all, including individuals with disabilities.
6. ***Data Privacy and Security:*** \
 Use AI for real-time monitoring of data security, ensuring compliance with regulations through automated checks.
7. ***Collaborative Partnerships:*** \
 Foster partnerships between educators, technologists, and policymakers. Leverage AI tools to facilitate collaborative problem-solving and decision-making.
8. ***Reflective Analytics:*** \
 AI can analyze historical data to provide insights on teaching effectiveness, supporting reflective practice and suggesting improvements.
9. ***Seamless Integration:*** \
 Use AI to integrate with other educational tools, enabling comprehensive analysis from various data sources without manual intervention.
10. ***Continuous Testing and Iteration:*** \
 Apply AI-driven A/B testing to refine the dashboard based on user feedback, ensuring it evolves to meet changing needs.





Context: Human-Centered {data-navmenu='Best Practices'}
=====================================

### Human Centered AI in Education

<p class="comment", style=\"color:darkblue;\">.. we found that the most prevalent type is visualisation or dashboards, accounting for 53% of the total of the included studies.</p>


![Human-centred learning analytics and AI in education: A systematic literature review](./IMG/human.png)



* [Citation](https://doi.org/10.1016/j.caeai.2024.100215)

Context: General:Education {data-navmenu='Best Practices'}
=====================================

### GenAI in Education

<p class="comment">We need to consider a lot of factors when reviewing Gen AI practices in Education in general and dashboards in specific. Here is a detailed meta review for Education.</p>

![[Citation](https://doi.org/10.3390/higheredu4010002)](./IMG/themes.png)

Context: General:LLM {data-navmenu='Best Practices'}
=====================================

### LLM challenges in Education

<p class="comment">[Citation](https://www.scirp.org/journal/paperinformation?paperid=137833)</p>

![LLM Survey](./IMG/LLM.jpeg)



Dashboard Examples {data-navmenu='Best Practices'}
=====================================

Here are a couple of examples of dashboards:

<hr>
* [Vizchat](https://github.com/LinxZhao/VizChat-pub)
* [Feedback copilot](https://www.sciencedirect.com/science/article/pii/S2666920X24000924)
* [Study buddy & teachers mate](https://ai4edu.eu/study-buddy-teacher-mate-platform/)
* [EDOER](https://www.edoer.eu)
* [health](https://arxiv.org/pdf/2501.09930)
* [Adjsting via reviewing Internal State](https://doi.org/10.48550/arXiv.2406.07882)
* [Commercial](https://teachmateai.com/free-tools)

<hr>


Topics {data-navmenu='Best Practices'}
=====================================

### Topics.

The purpose is to motivate further discussion with emphasis on the pedological aspects of the dashboard. We mention a number of technical aspects so that you do not have to.

Here are a number of mostly technical topics that ***need to be addressed*** through best practices. 

<details>
<summary> 
<p>Student Gen AI literacy</p>
</summary>
<p class="comment">How do we ensure that students understand the limitations of AI and how to interact with it?</p>
<hr>
* [Chatting with a Learning Analytics Dashboard:](https://doi.org/10.48550/arXiv.2411.15597)

<hr>
</details>


<details>
<summary> 
<p>OpenSource vs Closed Source</p>
</summary>
<p class="comment">
Should we have a preference for open-source solutions? If so, why and what actions are necessary to ensure consistency across the Educational sector? When should we abandon this best practice?
</p>
<hr>
* [A currently relevant question](https://arxiv.org/abs/2501.16403)

<hr>
</details>

<details>
<summary> 
<p>Censorship</p>
</summary> 
<p class="comment">Do we have to be careful how models are trained, their data inputs, and where the models are deployed? For example, when we use services that monitor and censor output.</p>
</summary>
<hr>
* [See Section 2.3](https://www.scirp.org/html/7-1763549_101420.htm)

<hr>
</details>

<details>
<summary> 
<p>Languages</p>
</summary>
<p class="comment">How many languages do our dashboards need to support? Does the quality of the dashboard output vary across languages? Does the dashboard need to auto-adapt to the language of any inputs?</p>
<hr>
* [Example from China](https://doi.org/10.1111/ejed.12749)
* [Medical language](https://arxiv.org/abs/2403.03640)
<hr>
* [Towards Safe Multilingual Frontier AI](https://arxiv.org/abs/2409.13708)
* [A multilingual benchmark](https://arxiv.org/abs/2303.12528)

<hr>
</details>

<details>
<summary> 
<p>DataSets</p>
</summary>
<p class="comment">Do we need open datasets for training? What is our exact definition of a dataset? How do we define quality, governance, and version control? Do we need to generate domain-specific datasets with relevant labelling and associated benchmarks? How can we share datasets openly?</p>
<hr>
* [Detecting hate](https://proceedings.neurips.cc/paper_files/paper/2023/hash/42f225509e8263e2043c9d834ccd9a2b-Abstract-Datasets_and_Benchmarks.html)
* [Curated datasets per industry](https://doi.org/10.48550/arXiv.2401.15544)
* [AI city challenge](https://doi.org/10.48550/arXiv.2404.09432)

<hr>
</details>

<details>
<summary> 
<p>Number of parameters</p>
</summary>
<hr>
<p class="comment">The number of parameters of a model roughly equates to CO2 consumption.
<br><br>
The parameters restrict whether we can run the model locally, whether the data stays local, computational effort, and latency. There is much to unpack about what technical details look like at first glance. However, it may have important implications for how we manage the deployment of models and ethics, privacy, and commercial engagement.
</p>
</hr>

</details>

<details>
<summary> 
<p>Prompt quality</p>
</summary>
<p class="comment">How do we define prompt quality? How do we benchmark prompt quality? Should we allow the AI to have an opinion about the prompt? Do we need prompting as part of teacher training?</p>
<hr>
* [AI literacy & Prompt Engineering](https://doi.org/10.1016/j.caeai.2024.100225)

<hr>
</details>

<details>
<summary> 
<p>Model/Prompt Hierarchy</p>
</summary>
<p class="comment">How do we define the hierarchy of models and prompts? How do we ensure that the  hierarchy is minimally biased?</p>
<hr>
* [Task.AI](https://doi.org/10.34133/icomputing.0063)

<hr>
</details>

<details>
<summary> 
<p>Defining model quality</p>
</summary>
<p class="comment">How do we define a minimum level of quality for a model?</p>
</details>

<details>
<summary> 
<p>Degree of hallucinations</p>
</summary>
<p class="comment">What is the acceptable level of hallucinations. When do we need humans in the loop?</p>
<hr>
* [Mitigating Halucinations. See textbox 1](https://doi.org/10.2196/59823)

<hr>
</details>

<details>
<summary> 
<p>EduLLM Ops - Required or Not?</p>
</summary>
<p class="comment">How do we maintain our values during the lifecycle of an LLM?</p>
</details>

<details>
<summary> 
<p>Accessibility</p>
</summary>
<p class="comment">What is the role of AI in making dashboards accessible to all intended audiences?</p>
</details>

<details>
<summary> 
<p>Model Persona</p>
</summary>
<p class="comment">Which model persona's are acceptable or how do we define which are not?</p>
</details>

<details>
<summary> 
<p>Size of Dashboard (e.g.: number of prompts)</p>
</summary>
<p class="comment">Do we scope the dashboards for specific tasks or generalize?</p>
</details>

<details>
<summary> 
<p>Cloud Policy</p>
</summary>
<p class="comment">There is much to be said on this subject.</p>
<hr>
* [When and where](https://doi.org/10.48550/arXiv.2104.10350) **Google**: *We are now optimizing where and when large models are trained.*

<hr>
</details>

Model Persona {data-navmenu='Best Practices'}
=====================================

### PERSONAS

<hr>
* [Adjsting via reviewing Internal State](https://doi.org/10.48550/arXiv.2406.07882)
* [Github Location](https://github.com/f/awesome-chatgpt-prompts/tree/main)
* [Interactive Repository](https://prompts.chat)

<hr>

```{r message=TRUE, echo=FALSE,warning=FALSE, results='asis'}
library(readr)
prompts <- read_csv("./DATA/prompts.csv", show_col_types = FALSE)

# 209 Prompts
idx <- c( 8,11,20,28,30,39,42,47,62,76,83,96,144,148,154,175,180)

for(i in idx){
  name <- prompts[i,1]
  prompt <- prompts[i,2]
  text <- paste("<details>\n\t<summary> 
          <p><b>",name,"</b></p>
          </summary>\n<hr><p style=\"color:darkblue;\">",prompt,"</p><hr></details>\n\n")
  cat(text)
}
```




The Team {data-navmenu=About}
=====================================

### Meet the Team

**Started:** 2024

***Target audience:*** students, teachers, ICTO'ers, I-Coaches, Policy makers.

* Priyanka Pereira, Universiteit Twente\
p.d.pereira@utwente.nl
* Alan Berg (HBO/WO – Hogeschool van Amsterdam en Universiteit van Amsterdam)\
a.m.berg@uva.nl
* Manuel Valle Torre (WO – Technische Universiteit Delft)\
M.ValleTorre@tudelft.nl

#### Alumini (2024)

* Anouschka van Leeuwen (projectleider)(WO - Universiteit Utrecht)
* Annie Slotboom (MBO – Graafschap)
* Symen van de Pas (MBO – InHolland)

#### Motivation for the team

Learning analytics has been a promise for years, but it has not yet been deployed on a large scale in the Netherlands. Are there any examples of how LA is delivering on its promise? And what does learning analytics bring to education, teachers and students? And what not? 
Insight into what effective methods are, which of them are also desirable (ethically responsible) and to which goals the use of LA has been proven to contribute, provides perspective and direction for the next phase in which the transformation with learning analytics (with the use of the National Learning Analytics Experimental Environment) ) can be used.

Dashboard {data-navmenu=About}
=====================================

### Details

***Author:*** Alan Berg: Npuls (LA Best and Worst Practices team).

R packages used in this dashboard:

* R Core Team (2024). _R: A Language and Environment for Statistical Computing_. R Foundation for Statistical Computing, Vienna,
  Austria. <https://www.R-project.org/>
* Aden-Buie G, Sievert C, Iannone R, Allaire J, Borges B (2023). _flexdashboard: R Markdown Format for Flexible Dashboards_. R package
  version 0.6.2, <https://CRAN.R-project.org/package=flexdashboard>.
* Gruber J, Weber M (2024). _rollama: Communicate with 'Ollama' to Run Large Language Models Locally_. R package version 0.2.0,
  <https://CRAN.R-project.org/package=rollama>.
* H. Wickham. ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, 2016.
* Massicotte P, South A (2023). _rnaturalearth: World Map Data from Natural Earth_. R package version 1.0.1,
*  Allaire J, Xie Y, Dervieux C, McPherson J, Luraschi J, Ushey K, Atkins A, Wickham H, Cheng J, Chang W, Iannone R (2024). _rmarkdown: Dynamic Documents for R_. R package version 2.29, <https://github.com/rstudio/rmarkdown>.
*Xie Y, Allaire J, Grolemund G (2018). _R Markdown: The Definitive Guide_. Chapman and Hall/CRC, Boca Raton, Florida. ISBN 9781138359338, <https://bookdown.org/yihui/rmarkdown>.
* Xie Y, Dervieux C, Riederer E (2020). _R Markdown Cookbook_. Chapman and Hall/CRC, Boca Raton, Florida. ISBN 9780367563837, <https://bookdown.org/yihui/rmarkdown-cookbook>. <https://CRAN.R-project.org/package=rnaturalearth>.
* Iannone R, Roy O (2024). _DiagrammeR: Graph/Network Visualization_. R package version 1.0.11,
  <https://CRAN.R-project.org/package=DiagrammeR>.

Introduction {data-navmenu=Extra}
=====================================

<p class="comment">In the extra section is random useful information we picked up along the way</p>

Code not direct output {data-navmenu=Extra}
=====================================


Column
-------------------------------------

### Code to generate images

**Write R code to generate an image of Europe with the names of the capital cities included**

> Expand the details in the prompt

1. [Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct)

**Advantage**

1. Code relies on Authoritative and deterministic sources of data (e.g.: in the packages).
1. Code is reproducible.

**Disadvantage**

1. Code is not interactive with human language.

Column{width=60%}
-------------------------------------

### Generated Image

![An almost correct map of Europe](./IMG/capitals.png){width=100%}
  
Think Formats {data-navmenu=Extra}
=====================================

Column
-------------------------------------

### Instructions

***Write in mermaid format a decision tree for making decisions at a meeting***

1. [Qwen2.5-Coder-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-32B-Instruct)
2. [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)

> Improve the prompt

<details>
<summary> 
<p>Initial Output for Live Editor (Note Line 1 modified to work with Editor)</p>
</summary>

```{bash,eval=FALSE, echo=TRUE}
flowchart TD
A[Start Meeting] --> B[Discuss Project Goals]
    B --> C{Feasibility Study Completed?}
    C -- Yes --> D{Budget Approved?}
    C -- No --> E[Conduct Feasibility Study]
    E --> B
    D -- Yes --> F{Team Availability?}
    D -- No --> G[Request Additional Budget]
    G --> H[Wait for Budget Approval]
    H --> D
    F -- Yes --> I{Market Research Completed?}
    F -- No --> J[Conduct Market Research]
    J --> F
    I -- Yes --> K[Proceed with Project]
    I -- No --> L[Delay Project]
    L --> M[Wait for Market Conditions]
    M --> I
    K --> N[End Meeting]
    L --> N
```
</details>

Test on this [online Editor](https://mermaid.live/)


Column
-------------------------------------

### Flowchart

```{r, fig.width=6, fig.height=5, fig.cap="Work in progress"}

library(DiagrammeR)

mermaid("
graph TD
    A[Start Meeting] --> B[Discuss Project Goals]
    B --> C{Feasibility Study Completed?}
    C -- Yes --> D{Budget Approved?}
    C -- No --> E[Conduct Feasibility Study]
    E --> B
    D -- Yes --> F{Team Availability?}
    D -- No --> G[Request Additional Budget]
    G --> H[Wait for Budget Approval]
    H --> D
    F -- Yes --> I{Market Research Completed?}
    F -- No --> J[Conduct Market Research]
    J --> F
    I -- Yes --> K[Proceed with Project]
    I -- No --> L[Delay Project]
    L --> M[Wait for Market Conditions]
    M --> I
    K --> N[End Meeting]
    L --> N
    ")

```

Improving your prompt {data-navmenu=Extra}
=====================================

### 

[Awesome Prompts](https://raw.githubusercontent.com/f/awesome-chatgpt-prompts/refs/heads/main/prompts.csv)

Let's refine the process of creating high-quality prompts together. Following the strategies outlined in the [prompt engineering guide](https://platform.openai.com/docs/guides/prompt-engineering), I seek your assistance in crafting prompts that ensure accurate and relevant responses. Here's how we can proceed: 

1. **Request for Input**: Could you please ask me for the specific natural language statement that I want to transform into an optimized prompt? 

2. **Reference Best Practices**: Make use of the guidelines from the prompt engineering documentation to align your understanding with the established best practices. 

3. **Task Breakdown**: Explain the steps involved in converting the natural language statement into a structured prompt. 

4. **Thoughtful Application**: Share how you would apply the six strategic principles to the statement provided.

5. **Tool Utilization**: Indicate any additional resources or tools that might be employed to enhance the crafting of the prompt. 

6. **Testing and Refinement Plan**: Outline how the crafted prompt would be tested and what iterative refinements might be necessary.  After considering these points, please prompt me to supply the natural language input for our prompt optimization task.

Favorite tools {data-navmenu=Extra}
=====================================

### 

Here are a list of technologies that we will be using in this demonstrator.





<details>
<summary> 
<p>**R markdown with flexdashboard**</p>
</summary>
Rapid Dashboard Development

* [Homepage](https://pkgs.rstudio.com/flexdashboard/)
* [Gallery](https://pkgs.rstudio.com/flexdashboard/articles/examples.html)
* [Rstudio IDE](https://posit.co/download/rstudio-desktop/)

</details>

<details>
<summary> 
<p>Ollama</p>
</summary>
Run models locally

* [Homepage](https://ollama.com)
* [Model list](https://ollama.com/search)
* [Download](https://ollama.com/download)

</details>

<details>
<summary> 
<p>Ngrok</p>
</summary>
Provide Access to locally run dashboard to share with a few colleges

* [Homepage](https://ngrok.com)

</details>

<details>
<summary> 
<p>AnythingLLM</p>
</summary>
Playground with RAG and tooling. Has a vector database out of the box and connectors include Ollama. Output from LLMS is displayed live. 

* [Homepage](https://anythingllm.com)

</details>

<details>
<summary> 
<p>HuggingFace</p>
</summary>
Call Models from the cloud. Interact with a hub of activity. Can chat with the models through the GUI

* [Homepage](https://huggingface.co)
* [Signup](https://huggingface.co/join)
* [Experiment with Models such as Deepseek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)
</details>