diff --git a/user-research/eventing-onboarding-research/Findings.md b/user-research/eventing-onboarding-research/Findings.md new file mode 100644 index 0000000..0619ac4 --- /dev/null +++ b/user-research/eventing-onboarding-research/Findings.md @@ -0,0 +1,107 @@ +## 5. Findings of the Research + +This section presents a comprehensive analysis of the key findings derived from both interview insights and survey results. The identified pain points have been carefully categorized based on their source, and corresponding actionable items have been proposed. To prioritize these actions, an Eisenhower Matrix has been employed, organizing tasks based on their urgency and importance. + +--- + +### 5.1. Combined Analysis of Pain Points and Actions + +#### 5.1.1. Lack of Prior Serverless, Kubernetes, and Docker Experience +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Users encountered difficulties with Knative Eventing due to a lack of prior experience with serverless technologies and Kubernetes. +- **Action**: + - Add a section in the documentation that clearly states the importance of Kubernetes and Docker knowledge. + - Provide links to existing Kubernetes learning resources to support the users' journey into Knative Eventing. + +#### 5.1.2. Abstract and Complex Documentation +- **Source**: Identified in both interviews and survey responses. +- **Issue**: The complexity and abstract nature of the documentation made it challenging for users. +- **Action**: + - Simplify the language in the documentation. + - Provide clear explanations of basic concepts. + - Add a glossary of terms and frequently used jargon. + +#### 5.1.3. Verification Difficulty +- **Source**: Identified in interviews. +- **Issue**: Difficulty verifying the correct setup of Knative Eventing after installation. +- **Action**: + - Create a simple example that installs a broker, a source, and an event display to verify the setup. + - Develop tools or scripts to help users verify installations. + - Include visual indicators or dashboards showing the status of components. + +#### 5.1.4. Lack of Practical Examples +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Users struggled with the absence of practical, real-life examples in the documentation. +- **Action**: + - Provide step-by-step tutorials and sample projects demonstrating the practical application of Knative Eventing. + +#### 5.1.5. Overwhelming Number of Repositories and Resources +- **Source**: Identified in interviews. +- **Issue**: The large number of repositories (70+) in the Knative organization was overwhelming, especially for beginners. +- **Action**: + - Consolidate documentation and resources into a clear, structured path for beginners. + +#### 5.1.6. Troubleshooting and Debugging Challenges +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Insufficient guidance in documentation for troubleshooting common issues. +- **Action**: + - Enhance the troubleshooting section with detailed guides, common issues, and resolutions. + - Include examples of error messages with explanations and resolutions. + +#### 5.1.7. Dependency on External Resources +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Many users relied on external resources like YouTube or third-party guides. +- **Action**: + - Curate and showcase high-quality external resources that complement official documentation. + +#### 5.1.8. Need for Mentorship and Community Support +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Insufficient resources for self-guided learning. +- **Action**: + - Move Q&A discussions from Slack to GitHub Discussions for a more permanent and searchable platform. + - Encourage expert participation in answering beginner questions. + +#### 5.1.9. Missing 'Why' and 'When' in Documentation +- **Source**: Identified in interviews. +- **Issue**: Documentation lacked clarity on 'why' and 'when' certain configurations are beneficial. +- **Action**: + - Include explanations of the reasoning behind different setups and configurations. + - Provide guidance on appropriate scenarios for specific approaches. + +#### 5.1.10. Consistency and Structure +- **Source**: Identified in both interviews and survey responses. +- **Issue**: Inconsistent naming conventions and steps across resources. +- **Action**: + - Standardize documentation structure and naming conventions. + - Align the format with Kubernetes documentation for improved navigation. + +--- + +### 5.2. Prioritization Using Eisenhower Matrix + +The Eisenhower Matrix categorizes tasks based on urgency and importance: + +#### High Urgency and Importance +1. Troubleshooting and Debugging Challenges +2. Lack of Practical Examples +3. Missing 'Why' and 'When' in Documentation +4. Lack of Prior Serverless, Kubernetes, and Docker Experience + +#### High Importance but Low Urgency +1. Abstract and Complex Documentation +2. Dependency on External Resources +3. Understanding Event-Driven Architecture + +#### High Urgency but Low Importance +1. Verification Difficulty + +#### Low Urgency and Importance +1. Overwhelming Number of Repositories and Resources +2. Need for Mentorship and Community Support +3. Consistency and Structure +4. Offer Training and Workshops + +This matrix provides a clear framework for prioritizing the actions needed to improve the Knative Eventing onboarding process, ensuring that the most critical pain points are addressed promptly, while less urgent issues are scheduled for later action. +![Eisenhower Matrix](./assets/Eisenhower-Matrix.png) + +--- diff --git a/user-research/eventing-onboarding-research/Future-Work.md b/user-research/eventing-onboarding-research/Future-Work.md new file mode 100644 index 0000000..85d811b --- /dev/null +++ b/user-research/eventing-onboarding-research/Future-Work.md @@ -0,0 +1,76 @@ +## 6. Next Steps and Future Sections + +The completion of this research marks a significant step towards enhancing the Knative Eventing onboarding experience. However, the journey doesn't end here. To provide a more comprehensive understanding and address all facets of this project, the following sections will be developed and included in the final report: + +--- + +### 6.1. Literature Review + +A detailed examination of existing research and documentation related to user onboarding in cloud-native environments, with a focus on serverless technologies, Kubernetes, and Knative. + +This section will: +- Compare Knative's current onboarding process with industry best practices. +- Identify areas where Knative aligns or diverges from these practices. +- Highlight opportunities for further alignment with best practices to improve onboarding effectiveness. + +--- + +### 6.2. Challenges Encountered During User Research + +A reflective analysis of the obstacles faced during the research process, including: +- Difficulties in participant recruitment. +- Limitations of the survey and interview methodologies. +- Issues encountered during usability testing. + +Understanding these challenges is crucial for interpreting the research findings and improving future research efforts. This section will provide transparency into the research process, ensuring stakeholders can build upon lessons learned. + +--- + +### 6.3. Recommendations + +Building on the findings of this report, the following implementation plans are suggested to enhance documentation, foster community support, and address key pain points identified through user feedback. These actionable strategies aim to ensure improvements are effectively executed while aligning with the broader goals of the Knative Eventing community. + +#### Enhancing Documentation +- **Action Item**: Utilize the existing GitHub project board to track specific enhancements to documentation. +- **Implementation**: + - Divide documentation improvements into prioritized sprints or phases. + - Assign tasks to contributors with set deadlines and validate updates during weekly Knative Eventing user group meetings. + - Establish a “Documentation Champion” role to oversee content accuracy, align updates with user feedback, and maintain resource quality. + +#### Fostering Community Support +- **Action Item**: Improve visibility and access to community support channels, particularly on GitHub Discussions. +- **Implementation**: + - Create a checklist for recurring topics that require community support. + - Designate experienced members to respond to beginner queries within set response-time guidelines. + - Maintain a categorized FAQ linked to these discussions to streamline access to essential information. + - Introduce a “Mentorship Pathway” to provide structured short-term mentoring for new contributors. + +#### Addressing Key Pain Points Through Training and Practical Examples +- **Action Item**: Enhance training efforts through curated live sessions or webinars addressing practical examples and common challenges. +- **Implementation**: + - Plan and schedule thematic training sessions using the project board. + - Focus on challenges like “Verification Difficulty” and “Understanding Event-Driven Architecture.” + - Record sessions for future reference to create sustainable learning resources. + - Develop an interactive “Learning Pathway” guide to help users navigate training materials based on experience level and interests. + +#### Continuous Improvement Through Targeted Feedback Collection +- **Action Item**: Schedule feedback sessions following significant documentation updates and training events. +- **Implementation**: + - Conduct bi-monthly focused feedback rounds to gather user input on recent changes. + - Use feedback forms and GitHub Issues to structure and track responses. + - Host community “Feedback Hours” for real-time discussions to rapidly iterate on arising issues. + +By adopting these strategies, the Knative community can enhance the onboarding process and establish an ongoing feedback loop, fostering a more inclusive and supportive environment. This will empower users to better engage with Knative Eventing and its ecosystem. + +--- + +### 6.4. References + +A comprehensive list of all sources cited in the report, including: +- Academic literature. +- Online resources. +- Tools used during the research. + +Proper citation of these references will be presented here and ensure the report is academically rigorous and provides a solid foundation for the recommendations made. + +--- diff --git a/user-research/eventing-onboarding-research/Introduction.md b/user-research/eventing-onboarding-research/Introduction.md new file mode 100644 index 0000000..10fe823 --- /dev/null +++ b/user-research/eventing-onboarding-research/Introduction.md @@ -0,0 +1,94 @@ +# Introduction + +This document introduces the **Knative Eventing Onboarding Research** project, conducted as part of the **LFX Mentorship Program**. It outlines the research objectives, challenges, and key findings aimed at improving the onboarding experience for new users of Knative Eventing. + +--- + +## 1. Executive Summary + +The aim of this research project is to enhance the onboarding experience for new users of **Knative Eventing**, a vital component of the Knative project that enables **event-driven architecture (EDA)** in cloud-native environments. Despite its powerful features, Knative Eventing poses challenges for newcomers due to its complexity and the learning curve associated with Kubernetes and event-driven systems. + +The primary objective is to identify the main barriers that new users face during their initial interactions with Knative Eventing. By understanding these pain points, the project seeks to develop improved documentation, practical onboarding materials, and streamlined processes to make the learning journey smoother for beginners. + +To achieve these goals, the research combines **surveys**, **interviews**, and **usability testing**. Engaging directly with the Knative community and recent users ensures that the findings are grounded in real-world experiences and needs. + +Ultimately, the project aims to produce resources that reduce the time it takes for new users to achieve competency and enhance their overall satisfaction with the onboarding process. Success can be measured through key performance indicators such as: +- **User satisfaction scores** +- **Onboarding completion rates** +- **Time to reach basic competency** + +This initiative is part of the **LFX Mentorship Program** and reflects a commitment to fostering an inclusive and supportive environment for new contributors to the open-source community. By improving the onboarding experience for Knative Eventing’s end-user, we aim to empower more developers to leverage its capabilities effectively and contribute to the growth of the Knative ecosystem. + +Upon completion of the research, the deliverables will be shared with the wider **CNCF community**, alongside the execution details of the project, to support other projects facing similar challenges. + + +### 1.1. Key Takeaways + +To provide a quick overview of the research outcomes, this section summarizes the most critical challenges and actionable insights identified during the study. These findings highlight key barriers faced by new Knative Eventing users and offer a foundation for targeted improvements to the onboarding process. + +- **Documentation Challenges**: Users struggled with abstract language, lack of practical examples, and insufficient troubleshooting guidance. +- **Knowledge Gaps**: Many users lacked prior experience with Kubernetes, serverless technologies, or event-driven architecture, increasing onboarding complexity. +- **Verification Issues**: A clear process to verify Knative Eventing installation is missing. +- **Overwhelming Resources**: The Knative repository structure is difficult to navigate for beginners. +- **Need for Community Support**: Beginners relied on external resources or informal mentoring due to gaps in official materials. +- **Improved Training Needs**: Real-life examples, recorded training sessions, and consistent naming conventions are crucial for easing the onboarding process. + +--- + +## 2. Research Overview and Goals + +### 2.1. Scope Definition + +The scope of this research project encompasses the entire onboarding experience for new users of Knative Eventing. The focus is on identifying and addressing the main challenges that users face during their initial interactions, improving the documentation, and enhancing the overall user experience. This includes: + +#### Focus Areas + +1. **User Research**: + Conducting surveys and interviews to gather insights from current and past users is vital for understanding their experiences and challenges with Knative Eventing onboarding. Surveys will collect quantitative data on user satisfaction and common issues, while in-depth interviews with a subset of users will provide qualitative insights. This approach helps identify specific pain points and gather comprehensive feedback, ensuring a thorough understanding of the onboarding experience. + +2. **Usability Testing**: + Observing new users as they engage with onboarding materials will highlight where they struggle. By creating realistic scenarios and monitoring their interactions, we can identify confusing instructions or difficult setup steps. Real-time communication with participants and recording sessions will provide valuable data to address the most critical issues in the onboarding process. + +3. **Documentation Improvement**: + Enhancing documentation is key to making the onboarding process more accessible. A thorough review of existing materials, combined with user feedback and usability testing results, will identify areas needing clarity and completeness. Revisions will improve accuracy and usability, while new resources such as step-by-step guides, video tutorials, and interactive demos will cater to different learning preferences. These improvements aim to simplify the learning curve for new users. + +4. **Community Engagement**: + Active participation in community forums, Slack channels, discussions on GitHub, and other platforms is essential for gathering continuous feedback and building support. Engaging with the Knative community ensures that improvements align with user needs and expectations. This collaborative approach fosters a sense of ownership and encourages ongoing contributions to enhance the onboarding experience. + +--- + +### 2.2. Limitations and Exclusions + +- **Exclusions**: This research will not cover advanced Knative Eventing features used by experienced users, nor will it delve into unrelated components of the Knative ecosystem. It will also exclude development changes to the Knative Eventing codebase itself, focusing instead on user-facing aspects. +- **Limitations**: The study is limited to the 12-week timeframe of the LFX Mentorship program, which may restrict the depth of certain analyses and the scope of iterative testing cycles. + +--- + +### 2.3. Identification of the Current Challenges + +Understanding the existing challenges is crucial for improving the onboarding experience. The research aims to: + +1. **Identify Common Pain Points**: + Determining the typical hurdles that new users face, such as difficulty understanding concepts, setting up environments, or navigating documentation, will demonstrate the points that need to be addressed within the context of this research. + +2. **User Feedback Collection**: + Using surveys and interviews to gather detailed feedback from users who have recently undergone the onboarding process will form the backbone of the improvement strategy. + +3. **Analyze Usability Issues**: + Observing where users struggle during usability testing and documenting specific instances of confusion or frustration will help to pinpoint critical pain points. + +--- + +### 2.4. Research Findings + +Effective documentation is critical to improving the accessibility and user experience of Knative Eventing's onboarding process. This initial insight serves as a foundation for enhancing the documentation structure, content clarity, and usability. Identified areas of improvement include: +- Addressing complex jargon. +- Adding real-world examples. +- Enhancing troubleshooting sections. +- Consolidating resources. + +These areas will be explored in detail in subsequent sections, with actionable strategies linked to specific pain points raised by users. + + +--- + diff --git a/user-research/eventing-onboarding-research/README.md b/user-research/eventing-onboarding-research/README.md new file mode 100644 index 0000000..97ed142 --- /dev/null +++ b/user-research/eventing-onboarding-research/README.md @@ -0,0 +1,61 @@ +# Knative Eventing Onboarding Research + +This repository contains the research report and supporting materials for improving the onboarding experience for **Knative Eventing**, conducted as part of the **LFX Mentorship Program**. The study identifies key challenges and proposes actionable recommendations for enhancing the onboarding process. + +--- + +## Table of Contents + +1. [Executive Summary](./Introduction.md##executive-summary) + - [Key Takeaways](./Introduction.md#key-takeaways) +2. [Research Overview and Goals](./Introduction.md#research-overview-and-goals) + - [Scope Definition](./Introduction.md#scope-definition) + - [Identification of the Current Challenges](./Introduction.md#identification-of-the-current-challenges) + - [Research Findings](./Introduction.md#research-findings) +3. [User Research and Analysis](./User-Research.md#user-research-and-analysis) + - [Survey Design](./User-Research.md#survey-design) + - [Interview Methodology](./User-Research.md#interview-methodology) + - [Usability Testing](./User-Research.md#usability-testing) +4. [Recruiting Participants](./Recruiting-Participants.md#recruiting-participants) + - [Community Outreach](./Recruiting-Participants.md#community-outreach) + - [Direct Invitations](./Recruiting-Participants.md#direct-invitations) +5. [Findings of the Research](./Findings.md#findings-of-the-research) + - [Combined Analysis of Pain Points and Actions](./Findings.md#combined-analysis-of-pain-points-and-actions) + - [Prioritization Using Eisenhower Matrix](./Findings.md#prioritization-using-eisenhower-matrix) +6. [Next Steps and Future Sections](./Future-Work.md#next-steps-and-future-sections) + - [Literature Review](./Future-Work.md#literature-review) + - [Challenges Encountered During User Research](./Future-Work.md#challenges-encountered-during-user-research) + - [Recommendations](./Future-Work.md#recommendations) + - [References](./Future-Work.md#references) + +--- + +## Project Management + +The research process was tracked using the [Eventing Onboarding Research Project Board](https://github.com/orgs/knative/projects/82/views/1). Tasks were organized into iterations to ensure progress tracking and timely deliverables. + +--- + +## Related Links + +- [Main Issue: Onboarding Experience Study for Knative Eventing (#130)](https://github.com/knative/ux/issues/130) +- [Knative UX Repository](https://github.com/knative/ux) + +--- + +## About the Author + +**Firat Bezir** +LFX Mentee, CNCF Mentorship Program +📧 [bezir.1@osu.edu](mailto:bezir.1@osu.edu) +🌐 [LinkedIn](https://www.linkedin.com/in/firatbezir/) | [GitHub](https://github.com/firatbezir?tab=overview&from=2024-06-01&to=2024-06-15) + +--- + +## How to Use This Repository + +1. Navigate through the **Table of Contents** to explore specific sections of the research. +2. Review the [Findings of the Research](./Findings.md#findings-of-the-research) and [Recommendations](./Future-Work.md#recommendations) for actionable insights. +3. Visit the [Project Board](https://github.com/orgs/knative/projects/2/views/1) to see how tasks were managed during this research. + +For any questions or feedback, feel free to reach out via the contact information provided above. diff --git a/user-research/eventing-onboarding-research/Recruiting-Participants.md b/user-research/eventing-onboarding-research/Recruiting-Participants.md new file mode 100644 index 0000000..0e69f8f --- /dev/null +++ b/user-research/eventing-onboarding-research/Recruiting-Participants.md @@ -0,0 +1,44 @@ +## 4. Recruiting Participants + +This section outlines the methods used to recruit participants for the research, focusing on engaging with the existing Knative community and reaching out to targeted individuals. These strategies ensured the inclusion of diverse perspectives and hands-on experiences to inform the study. + +--- + +### 4.1. Community Outreach + +Engagement with the existing Knative community was central to the participant recruitment strategy. The community, including developers, contributors, and users already involved with Knative Eventing, was utilized to identify individuals who could provide insightful feedback based on their experiences. + +#### Platforms Used: + +1. **Forums**: + Forums such as the Knative Eventing channels on Slack, the Knative Users Google Group, and other relevant discussion boards were used to recruit participants. These platforms enabled reaching users engaged in Knative Eventing's development and improvement. Slack, in particular, provided a space for real-time feedback and direct communication. Invitations to participate in the research, including surveys and interviews, were shared across these forums to build rapport and encourage participation. + +2. **GitHub**: + GitHub was used to identify active contributors to Knative Eventing. We analyzed [participants' contribution history](https://github.com/knative/eventing/graphs/contributors) to detect if they had contributed to Knative Eventing in the past six months. Additionally, [a dedicated GitHub project](https://github.com/orgs/knative/projects/82) was created for this research to track tasks iteratively. This allowed contributors to follow the project's progress transparently and gain insights into its objectives. + +3. **LinkedIn Posts**: + LinkedIn was used to reach a broader professional audience. Posts were made in relevant groups, and key Knative contributors and advocates were requested to publish these posts on their profiles. These posts invited users to participate in the research and share their onboarding experiences. This method leveraged the professional network and visibility of LinkedIn to attract participants who might not have been as active in other Knative-specific forums but still had valuable insights to offer. + +--- + +### 4.2. Direct Invitations + +To gather insights from users who had recent, hands-on experience with the onboarding process, direct invitations were sent to targeted individuals. + +#### Recruitment Methods: + +1. **New Users**: + Users who had recently completed the onboarding process were identified and contacted. This was done by: + - Tracking new contributions on GitHub (see the participant’s contribution history in [4.1 Community Outreach](#community-outreach) for details). + - Monitoring recent activity in Slack channels. + - Identifying posts in forums. + These users were likely to remember specific pain points and successes they encountered, providing valuable feedback. + +2. **Surveys and Interviews**: + Personalized invitations for participating in surveys and interviews were sent via email or direct messages on platforms such as Slack and GitHub. The importance of their participation and the potential impact of their feedback were explained to motivate a higher response rate. + +3. **Follow-ups**: + For users who initially responded positively but did not complete the survey or interview, follow-up messages were sent as gentle reminders. These messages emphasized the significance of their input and the impact it could have on improving the Knative Eventing experience for future users. + +--- + diff --git a/user-research/eventing-onboarding-research/User-Research.md b/user-research/eventing-onboarding-research/User-Research.md new file mode 100644 index 0000000..01259bd --- /dev/null +++ b/user-research/eventing-onboarding-research/User-Research.md @@ -0,0 +1,131 @@ +## 3. User Research and Analysis + +This section outlines the methods used to gather data from users, ensuring that the research approach is robust and comprehensive. Multiple methods were employed to capture a wide range of data, ensuring both depth and breadth in our analysis. Utilizing a combination of surveys, interviews, and usability testing was crucial because relying on a single approach would not provide the in-depth and nuanced insights needed to thoroughly understand the Knative Eventing onboarding process. + +To ensure the relevance and utility of the collected data, demographic information about participants was considered, focusing on proficiency, experience, and industry sector. + +--- + +### 3.1. Survey Design + +#### Objective + +Surveys were chosen to gather quantitative data due to their efficiency in collecting a large volume of responses. This method provided statistical insights into the prevalence of certain issues, user satisfaction levels, and common challenges faced during onboarding. By distributing surveys to both new and experienced users through various channels, we were able to capture a broad spectrum of perspectives, ensuring comprehensive data collection to identify patterns and trends in user experiences with Knative Eventing onboarding. + +#### Content + +1. **Technical Background**: + Questions were designed to understand the participant's level of experience across different technical domains such as front-end development, back-end development, DevOps, and serverless technologies. + - Example: + "Which of the following best describes your technical background? (Select all that apply and specify your level of experience for each)." + +2. **Familiarity with Serverless and Event-Driven Technologies**: + Questions focused on assessing participants' prior knowledge of serverless and event-driven architectures before using Knative Eventing. + - Example: + "How familiar were you with serverless technologies before using Knative Eventing?" + +3. **Onboarding Experience**: + Participants were asked about the duration and challenges they faced during the onboarding process, evaluating the effectiveness of the provided documentation and resources. + - Example: + "How long did it take you to complete the initial onboarding of Knative Eventing?" + +4. **Evaluation of Documentation**: + Questions evaluated the helpfulness of official and non-official Knative Eventing documentation during onboarding. + - Example: + "Which of the following sources did you find most helpful during your onboarding process with Knative Eventing?" + +5. **Challenges and Suggestions**: + Open-ended questions were included to allow participants to describe specific challenges and provide suggestions for improvement. + - Example: + "Based on your experience with Knative Eventing, what suggestions would you offer for improving the overall onboarding process?" + +#### Distribution + +The survey was distributed via email, Slack (as the primary community forum), and social media platforms like LinkedIn to target a diverse audience. + +#### Target Audience + +The survey targeted: +- **New users**: Individuals who had not started using Knative Eventing before. +- **Recent users**: Individuals who began using Knative Eventing within the past six months. +- **Experienced users**: To ensure a comprehensive view of onboarding challenges. + +--- + +### 3.2. Interview Methodology + +#### Objective + +Interviews were selected to obtain qualitative insights and detailed feedback. This method offered an opportunity for users to express their thoughts, experiences, and suggestions in their own words. Through open-ended questions, interviews allowed for a deeper exploration of user experiences, motivations, and pain points. By selecting participants representing diverse experience levels, the interview process ensured a holistic understanding of the onboarding process and uncovered nuanced insights that may not be captured through surveys alone. + +#### Content + +1. **Technical Background**: + Participants were asked to describe their prior experience with serverless technologies and software development to understand the context of their onboarding experience. + - Example: + "Please describe your experience related to serverless technologies or software development before using Knative Eventing." + +2. **Initial Encounter and Expectations**: + Participants were prompted to share their first impressions and expectations when starting with Knative Eventing. + - Example: + "Can you walk me through your initial encounter with Knative Eventing? What were your initial thoughts and expectations?" + +3. **Challenges and Documentation Feedback**: + The interview explored the specific challenges participants faced and gathered feedback on the clarity and effectiveness of the documentation. + - Example: + "Were there any aspects of the documentation that you found confusing or insufficient? If so, could you elaborate on your experience?" + +4. **Problem Resolution and Suggestions**: + Participants were asked to describe how they resolved challenges and to provide suggestions for improving the onboarding process. + - Example: + "When you faced difficulties, how did you resolve them? What would be the most helpful addition or change to Knative Eventing?" + +#### Participant Selection + +Participants were selected to ensure diversity in experience levels, including: +- Recent users just getting onboarded. +- Those with varying levels of technical expertise. + +#### Interview Process + +Interviews were conducted using platforms such as Google Meet, Microsoft Teams, and Zoom, with recordings and transcriptions securely stored for analysis. + +--- + +### 3.3. Usability Testing + +#### Objective + +Usability testing was employed to observe users interacting with the current onboarding materials, identifying specific pain points in the user experience. By creating realistic test scenarios and observing participants' interactions, we directly assessed the usability and effectiveness of the onboarding process. This method provided actionable feedback on areas for improvement, such as confusing UI elements or unclear instructions, contributing to a more user-friendly onboarding experience. + +#### Content + +1. **Test Scenarios**: + Scenarios were developed to reflect typical tasks new users need to complete, from initial setup to more advanced features of Knative Eventing. + - Example Task: + "Follow the instructions to install Knative Eventing on your local environment or a cloud provider and verify the installation by running a sample event." + +2. **Observation and Evaluation**: + Participants were observed while performing tasks, with their interactions recorded to identify usability issues. + - Example: + "How easy or difficult did you find the initial setup? Were there any specific points where you got stuck or confused?" + +3. **Post-Task Questions**: + After completing tasks, participants were asked to provide feedback on their experience, helping to pinpoint areas where improvements are needed. + - Example: + "Did the documentation or resources help you resolve any issues? If not, what additional information would have been helpful?" + +#### Evaluation Criteria + +The following metrics were used to evaluate usability: +- **Task Completion Rate**: Percentage of successfully completed tasks. +- **Time on Task**: Time taken to complete each task. +- **User Errors**: Number and types of errors made during tasks. +- **User Satisfaction**: Subjective feedback collected through post-task surveys or interviews. +- **Cognitive Load**: Assessment of the mental effort required to complete tasks. + +#### Ethical Considerations + +Participants were informed about the observation process, and consent was obtained. Confidentiality and anonymity were maintained for all recorded data. + +--- diff --git a/user-research/eventing-onboarding-research/assets/Eisenhower-Matrix.png b/user-research/eventing-onboarding-research/assets/Eisenhower-Matrix.png new file mode 100644 index 0000000..1a5e29b Binary files /dev/null and b/user-research/eventing-onboarding-research/assets/Eisenhower-Matrix.png differ