August 2024

For context, see the Annex which explains how we approach testing of Lawmaker.

Summary of v.17 testing

Version 17 includes several new features and a number of bug fixes: see the release note for details What's new in version 17? Our testing targeted those new features and improvements to ensure they worked correctly, and to check if they impacted existing Lawmaker functionality in any way

Each new feature and bug fix has been tested to ensure that is works as expected and does not adversely impact any existing Lawmaker functionality. In addition general regression testing has been carried out.

Tests were performed using a combination of newly created documents, documents imported from legislation.gov.uk, and existing real-world documents copied from the Production environment.

To test UK Bills functionality, we used in particular the Leasehold and Freehold Reform Bill, Renters' Reform Bill, and Digital Markets, Competition and Consumers Bill, along with the corresponding tabled amendments to those bills. Version 17 introduces a new feature which makes it possible to download and upload large lists of amendments to a bill version for testing; we made use of this new feature to perform tests on those bills using large numbers of real-world amendments.

We also imported UK Act content from legislation.gov.uk to test performance on large documents, including the Energy Act 2023 and Online Safety Act 2023.

Issues found during testing

1. We identified a bug which prevented images displaying in the Editor. This has been fixed in the final version 17 build.

2. We identified an existing bug with ‘remove reference tag’ function which also could cause problems with the updated ‘tag references’ function. The issue affected some references in documents containing line numbering mark-up. This has been fixed in the final version 17 build.

3. We observed that the automatic reference tagging function sometimes added additional reference tags around existing references; that has been fixed in the final build.

See below for more information on the specific testing outcomes related to new features and bug fixes in version 17.

Regression tests performed

End-to-end tests

The following end-to-end regression scripts and specfic feature scripts were run through by the Lawmaker team:

Additional UK Bill and Amendment testing

In addition to the general regression scripts, additional testing was carried out of UK Bill functionality using real data from previous sessions. This focused particularly on the auto-apply amendment functionality and ensuring that no regressions had been introduced by comparing the output of the auto-apply feature in the new version of Lawmaker with the output from the current version of Lawmaker.

PDF testing

PDFs were generated for a range of real and test documents and the outputs compared to PDFs generated with the current version of Lawmaker to identify and issues.

No issues were found.

New and improved feature testing

Automatic tagging of references

Version 17 includes a number of significant improvements to the automatic Tag x-refs function. This function will now identify both internal cross-references an and external references to existing legislation, and tag them.

Our testing focused on the new behaviour of the automatic tagging function in a variety of document types, using real-world documents. Where references were missed or tagged incorrectly, we worked with the development team to refine the tagging algorithm further. We have included those refinements in the final Version 17 deployment.test data.

Documents containing the newly tagged references were then used in end-to-end tests, to ensure that existing Lawmaker functionality is unaffected by the refinements to the automatic tagging function and the tags generated by that function. We did not identify any issues relating to the interaction of core Lawmaker functions and references as part of our end-to-end tests.

Referencing toolbar and the Edit ref modal

We tested the newly expanded referencing toolbar (which appears when double-clicking on a reference in the Editor) in all document types. We checked for potential user interface problems connected to the expanded toolbar, and we checked that initiating actions from the toolbar produced the same result as initiating those actions from the right-click menu.

We checked that the new reference toolbar had no effect on the performance of the Editor when opening larger documents.

User feedback testing on Staging helped us to identify an issue with the appearance and layout of the ‘Edit ref’ modal on smaller screens. We intend to fix this in the next minor Lawmaker release.

Generation of citation footnotes in SIs

Version 17 gives users the ability to automatically generate citation footnotes in SIs. We checked the content and format of these auto-generated footnotes using a large numbers number of real SI documents imported from legislation.gov.uk.In response to user feedback, we made a number of refinements to the wording and formatting of these footnotes during the pre-release testing period.

Improved sorting of amendments on the Amendment Manager tab

Version 17 replaces the ‘OoC Sort’ button on the Amendment Manager tab with a ‘Sort by Marshalled Order’ button that correctly sorts amendments in the table according to the order in which they will appear in official lists. We tested the new sorting using Commons, Lords, and Scottish Parliamentary Parliament amendments, and ensured that the ordering logic was correct. We identified and fixed an issue affecting sorting of We also tested using large numbers of real amendments (over 200). The fix has been included in the final version 17 build.

Church Measures

Version 17 now supports imported from the Production environment.

Quoted text

We tested adding Quoted Text elements in UK Bills, Scottish Bills, and SIs. We checked that existing Editor functions continued to work in documents containing these elements. We checked that PDF rendering and line numbering continued to work correctly in provisions which contained Quoted Text elements.

Church Measures

Version 17 adds supports for Church Measures documents. We tested the core Lawmaker functions in the Editor and on the Project tab in for Church Measures projects to ensure that users can draft and manage Measures in Lawmaker.

User feedback testing enabled us to identify some features which were not enabled for Church Measures (including the push/pull function and the ‘Submit to TNA’ action for enacted Measures). We worked with the development team to enable and test those features for Measures, and have included the necessary changes in the final version 17 release.

Significant issues found during testing

1. We identified a bug which prevented images displaying in the Editor. This has been fixed in the final version 17 build.

2. We identified a problem with the tag x-refs function and the ‘remove reference tag’ action which affected some references in documents containing line numbering. Where a reference extended over the end of a line, using the automatic tagging function or the ‘remove reference tag’ action (from the right-click menu or the floating toolbar) would sometimes remove both the XML tags and the textual content of the reference. Only the reference tags should be removed. This has been fixed in the final version 17 build.

3. The automatic reference tagging function sometimes added additional reference tags around existing references; that has been fixed in the final build.

Specific functions tested

Auto-apply

Inline amending

...

Uploading amendment list XML to create new amendments

Editing fragments of a schedule

Bug fix testing

The following bug fixes in version 17 were tested

SI/SSIs

Editor

PDFs

Amendments

Document management (Project tab, comparison, permissions etc.)

Annex - Overview of testing and development processes

This annex outlines how we implement quality control in relation to changes to the Lawmaker application during the development process and as part of the release process.

Testing during the development process

All new features and bug fixes are implemented by a multi-functional development team operating in an agile manner.

To ensure the quality of those changes, we implement the following steps:

1. The initial developer working on a feature or bug fix will, at the same time as writing the necessary code, write automated tests (“unit tests”) or modify existing ones, that can be run in future to ensure the feature/fix is still working as intended.

2. A second developer peer reviews the codes changes made by the first developer and the tests, before they are merged into the development environment.

3. When a change is merged into the development environment, all existing unit tests are run against the updated version to ensure that they still pass. If they don’t then the change will not be merged.

4. The new feature or fix is then tested by a QA engineer or another developer in the development environment to ensure it works as expected. A member of the Lawmaker service team tests feature/fix to ensure it meets the original specifications/expectations (this is effectively “user acceptance” of the change within the agile cycle).

If an issue is found at any of these stages, either it is passed back to an earlier stage in the process to be resolved or a separate task is created to resolve the issue.

Testing as part of the release process

Before any significant or minor release, a series of additional testing is carried out to ensure the quality of the release. The aim of this testing is to ensure that all new features and fixes work as expected when integrated together and also that no regressions in existing functionality have been introduced.

It is impossible to completely rule out any possibility of a regression being introduced somewhere but the testing aims to ensure that all business critical paths through the application work as expected and any defect introduced has only a minor impact on users.

So far as possible all issues found during release testing will be rectified before the release but work is prioritised as follows:

• if an issue affects a business critical paths then it will be fixed and retested before release,

• if it otherwise has a non-minor impact on users or could significantly erode user confidence in the robustness of the Lawmaker service then they will be fixed and retested before release,

• if it has a minor impact on users or is a pre-existing issue that is not being introduced by the release then it may be recorded in our issue-tracking system and addressed in a later release.

As part of the release process, the following steps are taken to ensure the quality of the release

1. An updated version of Lawmaker incorporating all changes is deployed to the Staging environment, which (apart from scale) is an exact replica of the Production environment.

2. Each new feature and bug fix included in the release is re-tested in the Staging environment.

3. A series of “end-to-end” test scripts are run through to ensure that the business-critical paths through the application work as expected. These scripts cover, in particular:

   1. Drafting and managing a Scottish Bill

   2. Drafting and managing amendments and the amendment process for a Scottish Bill

   3. Drafting and managing a UK Bill

   4. Drafting and managing amendments and the amendment process for a UK Bill

   5. The “ping pong” process for UK Bills

   6. Drafting and submitting an SI/SSI

4. A series of specific tests on particular areas of Lawmaker functionality are carried out, including:

   1. individual features and functions within the Editor,

   2. PDF generation,

   3. converting inline amendments to traditional amendments, and

   4. automatically applying amendments to a Bill.

5. Additional ad hoc testing by the Lawmaker Service Team also takes place in relation to any aspect of the application that is considered to be at particular risk of regression as a result of the changes made.

6. A series of automated load tests are carried out to ensure that the changes made in the new release do not have an impact on the robustness of the application.

7. Senior users within each user group are notified when the updated version of Lawmaker is deployed to the Staging environment and given the opportunity to view the changes and carry out ad hoc testing if they have capacity. Any feedback received from those users is reviewed and may either feed into further changes before the release (e.g. if a bug is identified) or will be recorded so that it can be addressed in future releases.