Here is a link to the google blog about the topic how many unit tests, integration and e2e tests there should be https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
There is a nice pyramid at the end of the blog post.

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests. The exact mix will be different for each team, but in general, it should retain that pyramid shape.

The blog post also defines both worlds and tries to describe what they should be used for, but I assume it's not descriptive enough for what we are searching.

Here is another link to the FAQ of protractor which tries to describe when it should be used, but it's even less descriptive than the blog post https://www.protractortest.org/#/faq#what-s-the-difference-between-karma-and-protractor-when-do-i-use-which-

IMO taken the definition for E2E test as simulation of a real user, an E2E should follow a defined scenario, like for example user creates a pool with a defined rule, or the user selects a specific pool and expects specific data that is provided through our backend.

What an E2E test should not do, IMO, is to just count rows or check if the column description names are fine - as the description names come from the frontend they can be tested as a unit test which is much faster. Just counting rows is omitting the content which can only be provided through E2E tests and therefore should not be omitted, but tested for it's completeness (as far as possible). Although just counting rows without looking at the content of them isn't simulating a user, as a user would read what the rows provide as information.

Nice pyramid:

As a good first guess, Google often suggests a 70/20/10 split: 70% unit tests, 20% integration tests, and 10% end-to-end tests.

I agree with the amounts of tests and also the order of importance, though, it feels odd to me to first write unit tests, then use Protractor for integration tests and then go back to the unit testing framework and use that for "E2E" testing, just because of performance reasons where there's no pressing or urgent need to worry about the performance.

I consider "counting rows" or "checking column names" not naturally to be a unit test. This is the interaction of many components. It's at the top of the pyramid and it's what a user see's at the end of the product and what a user would also look at, which makes it an End-to-End test in my opinion. In addition to that, the E2E tests will see and check those values backed by real data, where a unit tests will only mock the data to test a small component/unit.

The other argument for doing E2E testing in our unit test framework is performance. But the performance of running node in Jest and running node in Protractor isn't too different, except for the fact that E2E tests really request and render the page instead of just providing a fake DOM for testing. That makes, in summary, unit testing faster. But reading out and checking DOM objects after a page has been loaded isn't significantly slower. So if we follow the Pyramid by having 10% of our tests to be of those kind of checks, I don't even see a huge performance penalty.

The drawback of having E2E tests in a unit testing framework is the invitation to write unit tests and e2e tests at the same time, in the same framework. It basically invites developers to write unit tests and at the same time invites them to write E2E tests in the same framework. If the developer then doesn't add integration tests using protractor, we've followed the anti-pattern described in the google link mentioned:

Try to avoid these anti-patterns
[...]
Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.

What feels natural to me, in that order, is:

- Write unit/component tests in jasmine/jest, the unit testing framework
- Write integration tests in protractor, the e2e framework
- Write E2E tests in protractor, the e2e testing framework

For the sake of clarity and lack of a pressing need to do differently, I (currently) prefer to have those tests in our E2E testing framework, Protractor.

Patrick and I sat down and went through all the questions mentioned in the issue and came up with the following proposal (feedback is appreciated):

General introduction about testing and e2e/unit tests aka What are e2e/unit tests designed for?

E2E tests:
"Protractor is an end-to-end test framework for Angular and AngularJS applications. Protractor runs tests against your application running in a real browser, interacting with it as a user would." - http://www.protractortest.org/#/
It requires a working backend and tests the interation of all components of the application (Ceph, backend, frontend).
E2E tests are designed to mimic the behavior of the user when interacting with the application - for example when it comes to workflows like creating/editing/deleting an item. Also the tests should verify that certain items are displayed as a user would see them when clicking through the UI. Another good example could be a specific branding of the dashboard - the tests could help making sure that e.g. a specific logo or required colors are being displayed.

Angular Unit Tests:

Unit tests, as the name suggests, are tests for smaller units of the code. Those tests are designed for testing all kinds of Angulars' components (e.g. services, pipes etc.). They do not require a connection to the backend and hence are independ of it, although the expected result of the backend can be verified using only the frontend.
Data is either mocked or, in a simple case, contains a static input, a function call and an expected static output. More complex examples include the state of a component (attributes of the component class), that define how the output changes according to the given input.

Which e2e/unit tests are considered to be valid?

This is not easy to answer, but new tests that are written in the same way as already existing dashboard tests should generally be considered valid.

Unit tests should focus on the component to be tested.

E2E tests should focus on testing the functionality of the whole application. Approximately half of the E2E tests should verify the correctness of user visible elements.

How should an e2e/unit test look like?

Unit tests should focus on the described purpose and shouldn't try to test other things in the same `it` block.

E2E tests should contain a description that either verifies the correctness of a user visible element or a complete process like creation/validation/deletion of a pool.

What should an e2e/unit test cover?

E2E tests should mostly but not exclusively cover interaction with the backend. This way the interaction with the backend is utilized to write integration tests.

A unit test should mostly cover critical or complex functionality of a component (Angular Components, Services, Pipes, Directives, etc).

What should an e2e/unit test NOT cover?
- Avoid duplicate testing (do not write an e2e tests for what's already been covered as frontend-unit tests and vice versa)
It may not be possible to completely avoid an overlap.

Unit tests should not be used to extensively click through components and E2E tests shouldn't be used to extensively test a single component of Angular.

Best practices/examples/howtos
- Add examples of good tests we have in our code. TBD
- Add links to Design Guides and documents of best practices. TBD
- Link for Do's and Don'ts
- Add the pyramid! 70% Unit tests, 20,10

I agree with the amounts of tests and also the order of importance, though, it feels odd to me to first write unit tests, then use Protractor for integration tests and then go back to the unit testing framework and use that for "E2E" testing, just because of performance reasons where there's no pressing or urgent need to worry about the performance.

Naturally an valid E2E test can't be converted to an unit test. This is only possible if the E2E tests doesn't tests on the system scope and can be broken down into a integration scope.

I consider "counting rows" or "checking column names" not naturally to be a unit test. This is the interaction of many components.

Yes it is and because angular will import all needed frontend components to generate a fixture, which is in fact an integration of components and makes many (nearly any) frontend unit tests to an integration tests.

It's at the top of the pyramid and it's what a user see's at the end of the product and what a user would also look at, which makes it an End-to-End test in my opinion. In addition to that, the E2E tests will see and check those values backed by real data, where a unit tests will only mock the data to test a small component/unit.

I agree if the E2E tests checks the content of the values and does not just count them, it's a valid E2E test that integration between system <> ceph <> backend <> frontend usage, which can't and shouldn't be converted into an integration test.

The other argument for doing E2E testing in our unit test framework is performance. But the performance of running node in Jest and running node in Protractor isn't too different, except for the fact that E2E tests really request and render the page instead of just providing a fake DOM for testing. That makes, in summary, unit testing faster. But reading out and checking DOM objects after a page has been loaded isn't significantly slower. So if we follow the Pyramid by having 10% of our tests to be of those kind of checks, I don't even see a huge performance penalty.

Normally pure unit tests a extremely fast, but as said, the testing framework of angular will always integrate multiple components, which make them integration tests an therefor they will only be quite a lot faster, and with dev mode set to true a lot faster than E2E tests. But that's ok since the scope level differs, but that's wanted.

The drawback of having E2E tests in a unit testing framework is the invitation to write unit tests and e2e tests at the same time, in the same framework. It basically invites developers to write unit tests and at the same time invites them to write E2E tests in the same framework. If the developer then doesn't add integration tests using protractor, we've followed the anti-pattern described in the google link mentioned:

Try to avoid these anti-patterns
[...]
Hourglass. The team starts with a lot of unit tests, then uses end-to-end tests where integration tests should be used. The hourglass has many unit tests at the bottom and many end-to-end tests at the top, but few integration tests in the middle.

I fully agree, we should avoid writing E2E test if we can just group some units together to write an integration test (summed up the chapter about integration tests of the google blog post).

What feels natural to me, in that order, is:

- Write unit/component tests in jasmine/jest, the unit testing framework
- Write integration tests in protractor, the e2e framework

There is a big difference between E2E tests and integration tests. On the one side you test a hole system on the other you group specific units together. You can't do a specific integration of units and test how they play together even if something fails inside an E2E test. You can only write integration test with an unit tests framework. With E2E tests you always write system tests, as they always tests how the whole system plays together, and therefor you should make sure to validate Data that you would mock otherwise, to make sure the underlying layers haven't changed and are working as expected.

Types of testing¶

I'm now trying my best to visualize the different kind of tests we are discussing here, as there seems to be a missunderstanding. The short descriptions are taken from https://www.softwaretestinghelp.com/the-difference-between-unit-integration-and-functional-testing/ .

Unit tests¶

Short description¶

Unit testing means testing individual modules of an application in isolation (without any interaction with dependencies) to confirm that the code is doing things right.

Scope (inside same language)¶

[some class] > [some method]

Procedure¶

• Call "some method" of "some class" with mock data and expect a specific output.

Example¶

Test a static method

• Next number that is power of two with 82
• Expect 64.

Integration tests¶

Short description¶

Integration testing means checking if different modules are working fine when combined together as a group.

Scope (Combined units)¶

[n-Classes] > [n-Methods]

Procedure¶

• Init "some class" successfully (with all it's dependencies)
• Mock data streams if out of scope.
• Call "some method" of initialized class with mock data and expect a specific output.

Example¶

List 2 items in a table

• Mock 2 items
• Expect the fixture of the table two show a row count of 2

E2E tests / Functional tests¶

Short description¶

Functional testing means testing a slice of functionality in the system (may interact with dependencies) to confirm that the code is doing the right things.
Functional tests are related to integration tests, however, they signify to the tests that check the entire application’s functionality with all the code running together, nearly a super integration test.

Scope (System wide)¶

Running system

Procedure¶

• Mock user input
• Expect data (which in all other forms of testing is mocked)

Example¶

Pool creation

• Create a pool named "Jonny" with 128 PGs
• Expect the pool named "Jonny" on the pools table with 128 PGs

My feedback for Laura's and Patrick's comment.

[...]
It requires a working backend and tests the interation of all components of the application (Ceph, backend, frontend).

How about it rquires a fully functional system with ceph running and the dashboard activated with a running grafana and prometheus instance.
I wouldn't say frontend or backend here, as it's a system level scope.

Angular Unit Tests:

Unit tests, as the name suggests, are tests for smaller units of the code.

As we use the angular test mechanisms to use components and their fixtures in an virtual DOM, they aren't as simple as the name suggest. Most tests will be Angular integration tests.

E2E tests are designed [...] Another good example could be a specific branding of the dashboard - the tests could help making sure that e.g. a specific logo or required colors are being displayed.

If a logo is bound to a template or what color is displayed can be better tested inside an unit test as the logo and the colors are static values and only need the Angular scope to be verified.

Which e2e/unit tests are considered to be valid?

This is not easy to answer, but new tests that are written in the same way as already existing dashboard tests should generally be considered valid.

I'm not sure about the reasoning here, as we didn't had the documentation before. Nobody could have known about the outcome of it and couldn't know what the right approach was by just looking into our documentation. Therefor the written code can't automatically be seen as valid. Only because a tests passes doesn't makes it a valid system test. I also wan't to mention just because E2E tests are not valid anymore as the tests can be implemented as an integration tests, we should make sure to make them either valid E2E tests (in my favor) or convert them into integration tests, when we see such a case.

Unit tests should focus on the component to be tested.

A service is a good example to use unit tests only, most components have to be tested as integration tests, as they can't be initialized with their template without any other component.

E2E tests should focus on testing the functionality of the whole application. Approximately half of the E2E tests should verify the correctness of user visible elements.

You should not verify elements inside and E2E test you should use them and verify the data of your action.
(By just using them you are also verifying them as the test would fail if they are not there anymore)

How should an e2e/unit test look like?
[...]
E2E tests should contain a description that either verifies the correctness of a user visible element or a complete process like creation/validation/deletion of a pool.

A functional test should describe a scenario like "should create pool 'Jonny' with 128 PGs". As mentioned above it should not do unit specific things like checking if a form element exits. An E2E test should just use the form instead to archive something big, as it has the power to do so.

What should an e2e/unit test cover?

E2E tests should mostly but not exclusively cover interaction with the backend. This way the interaction with the backend is utilized to write integration tests.

E2E test will always test the interaction of the whole system, the only question is if the functional test uses this advantage of this scope, in order to make sure that the integration between all system layers is working.

I would remove the word of backend here as we are on a system level scope, therefore it will test the integration between the prometheus/grafana and the dashboard + dashboard and ceph + ceph and the distribution.

A unit test should mostly cover critical or complex functionality of a component (Angular Components, Services, Pipes, Directives, etc).

A unit isn't bound to it's complexity as all units together inside a component (integration on a component basis), reveal it's complexity. They should in the best case cover all code tracks, but it's better to have tests for the most complex behaviors than the least. Also every test writer should test that the test can fail through a false expectation.

What should an e2e/unit test NOT cover?
- Avoid duplicate testing (do not write an e2e tests for what's already been covered as frontend-unit tests and vice versa)
It may not be possible to completely avoid an overlap.

As explained before a valid functional test can't be reduced down to an integration/unit test. But unit tests shouldn't be deleted as soon as an E2E test tests a super-set of them. Like for example the form is working in the E2E tests than it's not the time to delete all unit tests that verify the form integrity. As unit test are needed to easily refactor code so that the form won't be broken through new changes.

We should also add a chapter what's the use of the one and the other, like unit tests give a lot more feedback to the developer and can be run during development all the time (TDD) and they can test edge cases. Functional tests are slow and test on a hole system, on a system that should not fail. The output in case of a failure is really vivid, but it's likely more critical than a unit test failure, but it can also be a random failure which mostly occur in E2E tests, as they have a far more complex infrastructure to run upon. If the failure isn't random it's a good indicator that somewhere else in ceph something was changed and the team hasn't noticed the change yet.

Unit tests should not be used to extensively click through components and E2E tests shouldn't be used to extensively test a single component of Angular.

I'm not sure why the term 'click' is used here as you can click on a fixture to trigger an action inside the unit test, which is pretty good as it tests if the template behaves as expected.
Maybe that's a better description what E2E tests should not do:
A functional test should use the dashboard as a user would and should not be aligned to the actual code as a unit test does.