Taquito Testing Strategies

We write and maintain tests so that we may check that the software we create will do what we think it should do and alert us when it doesn’t.

You can find more information on software testing philosophies in general here.

Assumptions

Taquito is a library. It offers tools. To test that Taquito is suitable for use, we must ensure that each tool the library provides does what it is supposed to. We also need to check that changes to support new features have not broken old features.

We use the following Test Heuristics to achieve these assurance goals. Taquito uses several of these in the CI/CD pipeline.

Unit tests

Unit Tests are detailed tests of simple software components at the atomic level. Taquito includes unit tests in each of its packages. Here is an example:

it('Public key returned by ledger device should be compressed adequately for tz1 before b58 encoding', () => {
  const buff = Buffer.from(
    '02063ed375b28dd2c1841138d4959f57b4a2715730e2e28fcda9144a19876dd3c6',
    'hex'
  );
  const compressbuff = compressPublicKey(buff, 0x00);
  const compressbuff2hex = Buffer.from(compressbuff).toString('hex');
  expect(compressbuff2hex).toEqual(
    '063ed375b28dd2c1841138d4959f57b4a2715730e2e28fcda9144a19876dd3c6'
  );
});

Sometimes a Unit Test will use a Mock to simulate interactions between the software and some external component.

We measure how comprehensive our unit test coverage is by running “test code coverage tools” that report on the lines of code that are not touched when running unit tests. We can verify that unit tests are effective with “mutation testing” described below.

Integration tests

Integration Tests look to ensure that multiple software components are working together. These components might be created by different teams or run on separate machines. The integration of various components can make these tests susceptible to random failures, but they will be rerun until they pass. Taquito testing routinely runs hundreds of integration tests daily on every pull request to the codebase.

Here is a simple example of an integration test. The test sends Taquito instructions to a live test node with the transactions processed on the blockchain. This test originates a contract on the chain with transfers and verifies that confirmation is received.

    it('Simple transfers with origination', async () => {
      const batch = await Tezos.batch()
        .withTransfer({ to: 'tz1ZfrERcALBwmAqwonRXYVQBDT9BjNjBHJu', amount: 0.02 })
        .withTransfer({ to: 'tz1ZfrERcALBwmAqwonRXYVQBDT9BjNjBHJu', amount: 0.02 })
        .withTransfer({ to: 'tz1ZfrERcALBwmAqwonRXYVQBDT9BjNjBHJu', amount: 0.02 })
        .withOrigination({
          balance: "1",
          code: ligoSample,
          storage: 0,
        })

      const op = await batch.send();
      await op.confirmation();
      expect(op.status).toEqual('applied')
    })

Code reviews

We do Code Reviews whenever a developer seeks to merge code. Other team members review it for correctness, accuracy, conformance with Taquito design principals, and suitability. This process will rapidly find problems that testing would either miss or take wasteful cycles to resolve. We will not merge code changes or new features unless they have been code reviewed and all requested changes have been either made or properly discussed.

Static code analysis

Static Code Analysis is run during the CI/CD cycle to do syntactic checks for errors in the code. Often a line marking a merge conflict or a violation of a coding format will cause a static analyzer to complain. During a CI/CD run, a Pull Request will be examined by CodeQL and Snyk.

End-to-End Tests

Taquito uses the Taquito Test dApp and the Live Code examples in the documentation as end-to-end tests. The tests exercise the entire software stack between the blockchain node and the user-facing interface. These tests show that all the components are working together, and are beneficial as they’re closer to a real use-case than what can be achieved with unit or integration tests. These tests are checked at each Taquito release.

Mutation tests

Mutation testing is a way to verify the effectiveness of unit tests. In addition to the code coverage of unit tests, we can check that the tests are resilient against all sorts of code changes. We all understand the intuition that if you change some code and no test fails, then the tests are missing something. Mutation testing tools allow us to implement this intuition and find missed test cases, confusing tests, missing assertions, etc. Taquito has been using Stryker to identify test mutations and systematically remove them from the code base. For details on how mutation testing works, please see: https://stryker-mutator.io/docs/.

Manual tests

When a user raises an issue, testers will verify the problem using manual methods. For Taquito, such testing could be:

• A small barebones Taquito script
• Checking a result with tezos-client
• Stepping through code with a debugger
• Rerunning scripts with small variations each time
• Other exploratory activities around the code base that are not fully scripted tests in the CI/CD.

Ledger Devices require manual testing as they have buttons that an operator must press to authorize signatures and transactions. There are emulators for Ledger Devices, but Taquito testing of ledger devices combines manual and scripted exploratory testing.

Security tests

Taquito has implemented some security tests in its integration test suite. These tests check for regressions in the Tezos code that could open known attack techniques.

The tests verify that a particular attack is impossible and that appropriate error messaging and exceptions occur when the tests try some well-known attacks.

Performance

ECAD DevOps maintains an extensive performance tracking and monitoring setup using Loki and Grafana, which generates alerts when specific performance parameters are out of band.

Managing Tezos protocol migrations with test nets

Each time Tezos changes protocol, there is a new test network, and old ones are deprecated. Contracts originated in a more senior test network must be originated again on the new test network. We have to update RPC content values and recreate Live Code Example contracts. So each protocol change requires an overhaul of some of the test assets to suit the new protocol.

The Taquito test suite runs in CI/CD against the current and next protocol testnets.

ECAD Devops maintains a suite of Tezos public nodes that the Tezos community can use. By supporting and monitoring these nodes, ECAD engineers have an overview and insights into the behaviour of these systems and can contribute to problem isolation, bug fixes and general troubleshooting; or specific test scenarios that require DevOps level node access.