PoC: Run E2E tests with WP Playground #62692

WunderBart · 2024-06-20T10:38:57Z

What?

A simple PoC for running E2E tests against a Playground instance (instead of wp-env).

How?

Checkout this branch.
Run npm install && npx playwright install to install deps.
Run npm run build to build Gutenberg.
Run node playground-server.js to start the test env via Playground.
Run the tests with WP_BASE_URL=http://127.0.0.1:9400/ npm run test:e2e.

This reverts commit 582c72c.

This reverts commit 12a8ac7.

WunderBart · 2024-06-21T15:36:37Z

@adamziel, many tests are failing because the database randomly disconnects, causing the app to throw 500 errors. I have yet to find a consistent way to reproduce the issue, but I'll continue working on it when I'm back (around July 2nd). Once this issue is resolved, the legit failures will provide valuable information on Playground's current limitations compared to wp-env.

It's been nice to see how easy it is to circumvent wp-env. Spinning up the WP Playground instance is really fast, too. 💪

cc: @griffbrad @dmsnell

swissspidy · 2024-07-02T08:14:05Z

FWIW for https://github.com/swissspidy/media-experiments I always use wp-now for running tests locally (as I can't use Docker) and I constantly run into the same issue with 500 errors. Curious to hear what you find out.

.github/workflows/end2end-test.yml

dmsnell · 2024-07-02T16:08:18Z

@WunderBart @swissspidy are you using a MySQL connection or relying on SQLite when you get the 500s? How do we know it's a database disconnect? I didn't think that is possible with SQLite, and I wonder also if we're seeing concurrency issues with multiple requests? Mostly the Playground handles requests serially, but I think it should be fine with concurrent requests as well.

swissspidy · 2024-07-02T18:37:35Z

Hmm I don't think it's a database issue in my case. Also no idea how to make it use MySQL, so I think it's just the default SQLite setup.

Just did another test and it's actually more a timeout than a proper error 500.

The server stops responding and in the terminal I get errors like this:

Error: PHP.run() failed with exit code 255 and the following output: 
    at _NodePHP.run (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72937:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async #dispatchToPHP (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72470:14)
    at async PHPRequestHandler.request (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72385:14)
    at async PHPBrowser.request (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72170:22)
    at async file:///Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@wp-now/wp-now/main.js:945:20 {
  output: {
    stdout: '<!DOCTYPE html>\n' +
      '<html lang="en-US">\n' +
      '<head>\n' +
      '\t<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n' +
      '\t<meta name="viewport" content="width=device-width">\n' +
      '\t\t',
    stderr: ''
  },
  source: 'request'
}

But I don't wanna hijack the discussion here :-)

github-actions · 2024-07-11T14:33:31Z

Warning: Type of PR label mismatch

To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.

Type-related labels to choose from: [Type] Automated Testing, [Type] Breaking Change, [Type] Bug, [Type] Build Tooling, [Type] Code Quality, [Type] Copy, [Type] Developer Documentation, [Type] Enhancement, [Type] Experimental, [Type] Feature, [Type] New API, [Type] Task, [Type] Technical Prototype, [Type] Performance, [Type] Project Management, [Type] Regression, [Type] Security, [Type] WP Core Ticket, Backport from WordPress Core, Gutenberg Plugin.
Labels found: Needs Author's Reply.

Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task.

This reverts commit bfbc5fa.

This reverts commit df7d403.

WunderBart · 2024-10-06T17:14:01Z

@WunderBart @swissspidy are you using a MySQL connection or relying on SQLite when you get the 500s? How do we know it's a database disconnect? I didn't think that is possible with SQLite, and I wonder also if we're seeing concurrency issues with multiple requests? Mostly the Playground handles requests serially, but I think it should be fine with concurrent requests as well.

The db disconnected is what's being thrown every time that 500 happens:

@dmsnell @swissspidy @adamziel, I'm pretty sure it's a Playground memory leak / CI resources issue. It's not tied to any tests specifically and to any specific Playwright action (although Playwright might also be leaking 🤷). To confirm that, I tried running a smaller number of tests per job, and the ones that the suites usually fail at pass if run in smaller batches. For example, here are three iterations, each running more tests from the previous one, finally (consistently) failing at iteration 3:

FWIW, I did observe the server logs and they weren't any helpful. All the requests end with 200 until suddenly it's 500.

It also seems to be related to what we're experiencing in Woo where we're running with wp-env, which suggests it might be a Playwright/PHP memory issue (?): woocommerce/woocommerce#51658 (comment)

I'm kinda out of ideas on how to debug this further, also because I need a break from this issue. 😅 Any help here would be much appreciated! 🙏

adamziel · 2024-10-07T17:31:02Z

@wojtekn does that happen at all in Studio?

adamziel · 2024-10-07T17:35:55Z

@WunderBart wonderful explorations, thank you! Let's extract as much debug information as we can. A few ideas:

Enabling all the debugging constants
Upload Playwright trace with, hopefully, some specific PHP errors on the screen
Upload WordPress debug.log as an artifact
Upload the WordPress directory as an artifact – hopefully the failure would still be reproducible after downloading it

Also, are you running @wp-playground/cli/wp-now? If yes, and we're indeed dealing with some sort of memory leak, you could kill and restart the server and then re-run the test. However, if you're running the browser version of Playground, there isn't anything similar we could do.

Also let's link this with this related issue: WordPress/wordpress-playground#1596

wojtekn · 2024-10-08T13:25:36Z

@wojtekn does that happen at all in Studio?

@adamziel No, I didn't spot such an issue in Studio.

WunderBart · 2024-10-08T14:17:25Z

Thanks for the feedback, @adamziel!

@WunderBart wonderful explorations, thank you! Let's extract as much debug information as we can. A few ideas:

Enabling all the debugging constants

Upload Playwright trace with, hopefully, some specific PHP errors on the screen

Upload WordPress debug.log as an artifact

Upload the WordPress directory as an artifact – hopefully the failure would still be reproducible after downloading it

Also, are you running @wp-playground/cli/wp-now? If yes, and we're indeed dealing with some sort of memory leak, you could kill and restart the server and then re-run the test. However, if you're running the browser version of Playground, there isn't anything similar we could do.

I'm running with @wp-playground/cli server. I thought about making a custom retry mechanism for restarting the server, but I believe it's more important here to fix the 500 issue so we can safely run any number of tests with a vanilla setup.

An example trace.zip isn't really that helpful - it just shows that the test fails at navigation or plugin activation because of the 500:

The server logs around the 500 aren't that helpful either. Everything is peachy until a sudden 500 is thrown (tried with different PHP versions FWIW). Here's a snippet from around the error line:

172.18.0.1 - - [06/Oct/2024:14:53:36 +0000] "GET /wp-json/wc/store/v1/cart?_locale=user HTTP/1.1" 200 2093 "http://localhost:8889/wp-admin/site-editor.php?postId=woocommerce%2Fwoocommerce%2F%2Forder-confirmation&postType=wp_template&canvas=edit" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.18 Safari/537.36"
172.18.0.1 - - [06/Oct/2024:14:53:37 +0000] "POST /wp-json/wp/v2/users/me?_locale=user HTTP/1.1" 200 5539 "http://localhost:8889/wp-admin/site-editor.php?postId=woocommerce%2Fwoocommerce%2F%2Forder-confirmation&postType=wp_template&canvas=edit" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.18 Safari/537.36"
172.18.0.1 - - [06/Oct/2024:14:53:37 +0000] "PUT /wp-json/wp/v2/plugins/woocommerce-blocks-test-plugins/register-product-collection HTTP/1.1" 500 438 "-" "Playwright/1.46.1 (x64; ubuntu 20.04) node/20.17 CI/1"
127.0.0.1 - - [06/Oct/2024:14:53:38 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.54 (Debian) PHP/7.4.33 (internal dummy connection)"
172.18.0.1 - - [06/Oct/2024:14:53:38 +0000] "GET /wp-json/wp/v2/plugins HTTP/1.1" 500 2719 "-" "Playwright/1.46.1 (x64; ubuntu 20.04) node/20.17 CI/1"
127.0.0.1 - - [06/Oct/2024:14:53:39 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.54 (Debian) PHP/7.4.33 (internal dummy connection)"
172.18.0.1 - - [06/Oct/2024:14:53:39 +0000] "GET /wp-json/wp/v2/plugins HTTP/1.1" 500 2719 "-" "Playwright/1.46.1 (x64; ubuntu 20.04) node/20.17 CI/1"
172.18.0.1 - - [06/Oct/2024:14:53:39 +0000] "GET /wp-json/wp/v2/plugins HTTP/1.1" 500 2719 "-" "Playwright/1.46.1 (x64; ubuntu 20.04)

Maybe I'm just missing something obvious there, though! 🤞

Do we have any DB experts in this context who could help inspect the issue from that side? 😅

WunderBart added 2 commits June 20, 2024 12:15

modify config to run against Playground

0399391

upload e2e blueprint

e226d20

WunderBart force-pushed the try/run-e2e-tests-against-playground branch 2 times, most recently from 26f785c to f60c136 Compare June 20, 2024 13:38

WunderBart added 2 commits June 20, 2024 15:56

try running e2e tests w/ Playground in CI

467a732

make the test report verbose

5416734

WunderBart force-pushed the try/run-e2e-tests-against-playground branch from f60c136 to 5416734 Compare June 20, 2024 13:59

WunderBart changed the title ~~PoC: Run E2E tests against WP Playground~~ Jun 20, 2024

WunderBart added 7 commits June 20, 2024 16:27

run single suite for testing

12a8ac7

try checking out PG repo aside

582c72c

Revert "try checking out PG repo aside"

2991126

This reverts commit 582c72c.

restore global setup for pre-auth

187acdf

add missing dir change

4e2de6d

Revert "run single suite for testing"

a54920c

This reverts commit 12a8ac7.

disable retries to finish faster avoid flaky reports

a34384e

dmsnell reviewed Jul 2, 2024

View reviewed changes

.github/workflows/end2end-test.yml Outdated Show resolved Hide resolved

dmsnell reviewed Jul 2, 2024

View reviewed changes

.github/workflows/end2end-test.yml Outdated Show resolved Hide resolved

adamziel assigned WunderBart Jul 3, 2024

adamziel mentioned this pull request Jul 10, 2024

Intermittent SQLite Database Disconnects WordPress/wordpress-playground#1596

Open

adamziel added the Needs Author's Reply label Jul 11, 2024

adamziel removed the Needs Author's Reply label Jul 11, 2024

adamziel mentioned this pull request Jul 16, 2024

Performance testing next steps (adoption) WordPress/performance#1093

Open

adamziel added the Needs Author's Reply label Jul 16, 2024

swissspidy mentioned this pull request Jul 19, 2024

Blueprint support swissspidy/wp-performance-action#71

Closed

carstingaxion mentioned this pull request Aug 21, 2024

NEW e2e environment using playground cli GatherPress/gatherpress#810

Open

5 tasks

adamziel removed the Needs Author's Reply label Oct 3, 2024

Add db connection retry

df7d403

WunderBart force-pushed the try/run-e2e-tests-against-playground branch from 890ed10 to df7d403 Compare October 3, 2024 14:48

Wait for the server to be fully provisioned

49cfa8d

WunderBart force-pushed the try/run-e2e-tests-against-playground branch from bbc4e3a to 5603487 Compare October 3, 2024 16:48

Provision server with node

1e0155b

WunderBart force-pushed the try/run-e2e-tests-against-playground branch from 5603487 to 1e0155b Compare October 3, 2024 17:50

WunderBart added 14 commits October 3, 2024 20:55

Ignore the Downloading ... log noise

6dca9a6

Don't print progress logs

1613cbb

Debug pw:api

e003621

Try older php

b294df5

Fix log exclusion

4fe28bb

Try POST instead of PUT for plugin activation

bfbc5fa

Revert "Try POST instead of PUT for plugin activation"

80e61be

This reverts commit bfbc5fa.

disable pw:api logs

9ca0fbb

Revert "Add db connection retry"

ef07ba4

This reverts commit df7d403.

try: Run only one shard

8549d76

try: Make shards smaller

19d223f

tmp: run just 1 suite

d92a140

tmp: run 2 suites :)

3e48cba

try: run 3 suites :))

15689a1

adamziel added the Needs Author's Reply label Oct 8, 2024

adamziel removed the Needs Author's Reply label Oct 8, 2024

adamziel mentioned this pull request Oct 16, 2024

[Documentation] Identify the most impactful improvements in Playground learning resources WordPress/wordpress-playground#1812

Open

adamziel added the Needs Author's Reply label Oct 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC: Run E2E tests with WP Playground #62692

PoC: Run E2E tests with WP Playground #62692

WunderBart commented Jun 20, 2024 •

edited

Loading

WunderBart commented Jun 21, 2024

swissspidy commented Jul 2, 2024

dmsnell commented Jul 2, 2024

swissspidy commented Jul 2, 2024

github-actions bot commented Jul 11, 2024 •

edited

Loading

WunderBart commented Oct 6, 2024 •

edited

Loading

adamziel commented Oct 7, 2024

adamziel commented Oct 7, 2024 •

edited

Loading

wojtekn commented Oct 8, 2024

WunderBart commented Oct 8, 2024 •

edited

Loading

PoC: Run E2E tests with WP Playground #62692

Are you sure you want to change the base?

PoC: Run E2E tests with WP Playground #62692

Conversation

WunderBart commented Jun 20, 2024 • edited Loading

What?

How?

WunderBart commented Jun 21, 2024

swissspidy commented Jul 2, 2024

dmsnell commented Jul 2, 2024

swissspidy commented Jul 2, 2024

github-actions bot commented Jul 11, 2024 • edited Loading

WunderBart commented Oct 6, 2024 • edited Loading

adamziel commented Oct 7, 2024

adamziel commented Oct 7, 2024 • edited Loading

wojtekn commented Oct 8, 2024

WunderBart commented Oct 8, 2024 • edited Loading

WunderBart commented Jun 20, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading

WunderBart commented Oct 6, 2024 •

edited

Loading

adamziel commented Oct 7, 2024 •

edited

Loading

WunderBart commented Oct 8, 2024 •

edited

Loading