Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addon-Test: Report component file tests to the frontend #29388

Open
wants to merge 1 commit into
base: unified-ui-testing
Choose a base branch
from

Conversation

valentinpalkovic
Copy link
Contributor

@valentinpalkovic valentinpalkovic commented Oct 17, 2024

Closes N/A

What I did

I have changed the reporting mechanism for story files, which cannot be executed by Vitest. For example, if the preview.js file throws an error or a story file has issues to process an import and throws.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

  • stories
  • unit tests
  • integration tests
  • end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

  • Add or update documentation reflecting your changes
  • If you are deprecating/removing a feature, make sure to update
    MIGRATION.MD

Checklist for Maintainers

  • When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts

  • Make sure this PR contains one of the labels below:

    Available labels
    • bug: Internal changes that fixes incorrect behavior.
    • maintenance: User-facing maintenance tasks.
    • dependencies: Upgrading (sometimes downgrading) dependencies.
    • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
    • cleanup: Minor cleanup style change. Will not show up in release changelog.
    • documentation: Documentation only changes. Will not show up in release changelog.
    • feature request: Introducing a new feature.
    • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
    • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name before after diff z %
createSize 0 B 0 B 0 B - -
generateSize 78.7 MB 78.7 MB 0 B 0.24 0%
initSize 147 MB 147 MB 48 B -0.21 0%
diffSize 68.3 MB 68.3 MB 48 B -0.21 0%
buildSize 7.1 MB 7.1 MB 52 B 0.35 0%
buildSbAddonsSize 1.79 MB 1.79 MB 40 B 0.75 0%
buildSbCommonSize 195 kB 195 kB 0 B - 0%
buildSbManagerSize 1.85 MB 1.85 MB 12 B -0.1 0%
buildSbPreviewSize 271 kB 271 kB 0 B -1.27 0%
buildStaticSize 0 B 0 B 0 B - -
buildPrebuildSize 4.1 MB 4.1 MB 52 B 0.5 0%
buildPreviewSize 3 MB 3 MB 0 B 0.26 0%
testBuildSize 0 B 0 B 0 B - -
testBuildSbAddonsSize 0 B 0 B 0 B - -
testBuildSbCommonSize 0 B 0 B 0 B - -
testBuildSbManagerSize 0 B 0 B 0 B - -
testBuildSbPreviewSize 0 B 0 B 0 B - -
testBuildStaticSize 0 B 0 B 0 B - -
testBuildPrebuildSize 0 B 0 B 0 B - -
testBuildPreviewSize 0 B 0 B 0 B - -
name before after diff z %
createTime 19.3s 24.7s 5.3s 2.03 🔺21.8%
generateTime 20.1s 21s 850ms -0.43 4%
initTime 14s 13.1s -870ms -0.82 -6.6%
buildTime 8.5s 8s -440ms -1.15 -5.4%
testBuildTime 0ms 0ms 0ms - -
devPreviewResponsive 5.6s 6.8s 1.1s 1.42 🔺16.8%
devManagerResponsive 3.8s 4.1s 271ms 0.3 6.5%
devManagerHeaderVisible 513ms 609ms 96ms 0.14 15.8%
devManagerIndexVisible 545ms 642ms 97ms 0.15 15.1%
devStoryVisibleUncached 926ms 1s 90ms 0.5 8.9%
devStoryVisible 546ms 641ms 95ms 0.13 14.8%
devAutodocsVisible 519ms 499ms -20ms -0.62 -4%
devMDXVisible 453ms 525ms 72ms -0.09 13.7%
buildManagerHeaderVisible 508ms 555ms 47ms -0.4 8.5%
buildManagerIndexVisible 568ms 598ms 30ms -0.34 5%
buildStoryVisible 569ms 599ms 30ms -0.58 5%
buildAutodocsVisible 500ms 459ms -41ms -1.38 🔰-8.9%
buildMDXVisible 449ms 459ms 10ms -1.01 2.2%

Greptile Summary

This pull request enhances error reporting and status updates for component tests in Storybook, focusing on improved handling of test failures and crashes.

  • Updated TestingModuleCrashReportPayload type to include an error object with a message property
  • Enhanced StorybookReporter class to provide more detailed reporting of test suite failures
  • Modified SidebarBottom component to handle crash and progress reports more effectively
  • Improved error handling in boot-test-runner.ts for better alignment with the new error reporting structure
  • Added telemetry for test discrepancies in preset.ts

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6 file(s) reviewed, 11 comment(s)
Edit PR Review Bot Settings | Greptile

code/addons/test/src/manager.tsx Show resolved Hide resolved
Comment on lines +102 to 103
} else if (failed && !errorMessage) {
message = 'Component tests failed';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: This condition might not cover all failure scenarios

code/addons/test/src/manager.tsx Show resolved Hide resolved
Comment on lines +172 to +173
testResults: TestResult[];
error?: { message: 'string'; name: 'string' };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

syntax: The 'string' type should not be in quotes

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valentinpalkovic Looks like Greptile is correct. I would've totally overlooked it myself.

code/addons/test/src/node/reporter.ts Show resolved Hide resolved
code/addons/test/src/node/reporter.ts Show resolved Hide resolved
code/addons/test/src/node/reporter.ts Show resolved Hide resolved
code/addons/test/src/node/reporter.ts Show resolved Hide resolved
Copy link

nx-cloud bot commented Oct 17, 2024

☁️ Nx Cloud Report

CI is running/has finished running commands for commit 204db5d. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution


✅ Successfully ran 1 target

Sent with 💌 from NxCloud.

@valentinpalkovic valentinpalkovic force-pushed the valentin/report-component-test-errors branch from 7cb691a to c04d44a Compare October 17, 2024 13:07
@valentinpalkovic valentinpalkovic force-pushed the valentin/report-component-test-errors branch from c04d44a to 204db5d Compare October 17, 2024 13:22
Comment on lines +172 to +173
testResults: TestResult[];
error?: { message: 'string'; name: 'string' };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valentinpalkovic Looks like Greptile is correct. I would've totally overlooked it myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants