Version 18.0.0 Test Report
- Edward Griffiths
December 2024
For context, see the Annex to this report which explains how we approach testing of Lawmaker.
Summary of v.18 testing
Version 18 includes several new features and a number of bug fixes: see the release note for details What's new in version 18?
Each new feature and bug fix has been tested to ensure that is works as expected and does not adversely impact any existing Lawmaker functionality. In addition general regression testing has been carried out.
Tests were performed using a combination of newly created documents, documents imported from legislation.gov.uk, and existing real-world documents copied from the Production environment.
To test UK Bills functionality, we used in particular the Renters (Reform) Bill, the Employment Rights Bill, the Automated Vehicles Bill, and the Football Governance Bill along with the corresponding tabled amendments to those bills.
We also imported UK Act content from legislation.gov.uk to test performance on large documents, including the Energy Act 2023.
Issues found during testing
We identified a bug which caused too much text to be wrapped in inline style tags when selecting text before a Quoted Structure and applying bold, underline, or italic styling to it. This has been fixed in the final version 18 build.
We identified an issue which prevented the assignment of members' names to Motions in LoDAs at Ping Pong stage. This has been fixed in the final version 18 build.
A false-positive document check was identified which affects adjacent Quoted Structures in version 18. We intend to fix this in a future support release.
See below for more information on the specific testing outcomes related to new features and bug fixes in version 18.
Regression tests performed
End-to-end tests
The following end-to-end regression scripts and specfic feature scripts were run through by the Lawmaker team:
Script | Test result | Notes |
|---|---|---|
UK Bill Test Script | Pass |
|
UK Amendments Test Script | Pass |
|
UK Ping Pong test script | Pass | Identified a problem with adding Members’ names as proposers and supporters on Motions; this has been fixed in the final v.18.0 build. |
Editor Test Matrix script | Pass |
|
Scottish Bill test script | Pass |
|
Scottish Amendments test script | Pass |
|
Additional UK Bill and Amendment testing
In addition to the general regression scripts, additional testing was carried out of UK Bill functionality using real data from current and previous parliamentary sessions. This focused particularly on the auto-apply amendment functionality to ensure that the majority of amendment scenarios could be applied, and that the same scenarios which are currently supported in version 17.2 are still supported in version 18.0.
Some known limitations of auto-apply were observed during testing; these will be fixed in a future support release.
(Note that support for auto-application of amendments which leave out, insert, or substitute whole rows in tables will be added in version 18.1)
Bill title | Amendments tested | Notes |
|---|---|---|
Employment Rights Bill (House of Commons, As Introduced) | 102 out of 114 amendments applied Scenarios tested:
|
|
Renters (Reform) Bill (House of Commons, As Introduced) | 166 out of 176 amendments applied Scenarios tested:
| Some amendments failed to apply because they were leaving out the last line of a Quoted Structure immediately before a Schedule cross-heading. We will fix this limitation in a future support release. |
Automated Vehicles (House of Lords, As Introduced) | 53 out of 55 amendments applied Scenarios tested:
|
|
Football Governance (House of Lords, As Introduced) | 171 out of 199 amendments applied Scenarios tested:
|
|
Additional Scottish Bill and Amendment testing
Auto-apply was also tested using the following Scottish bills and a selection of lodged amendments:
Bill title | Amendments tested | Notes |
|---|---|---|
Bankruptcy and Diligence (Scotland) Bill (As Introduced) | 27 out of 30 applied Scenarios tested:
|
|
Regulation of Legal Services (Scotland) Bill (As Introduced) | 196 out of 214 applied In order to check Editor performance in a bill containing a large number of applied amendments, we tested auto-apply using the latest batch of lodged government amendments to this bill. |
|
PDF testing
PDFs were generated for a range of real and test documents and the outputs compared to PDFs generated with the current version of Lawmaker to identify any unexpected differences.
No issues were found.
New and improved feature testing
Auto-apply and inline amending
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4486 |
| Auto-apply amendments to any bill version which is currently open in the Editor | Pass |
LDAPP-4475 | LMS-2742 | Auto-apply amendments which insert a Cross-heading, Part, or other grouping level in the Body or Schedules of a Bill | Pass; but note that amendments inserting cross-headings into Quoted Structures which don’t contain those groupings are not currently applied. |
LDAPP-4471 | LMS-668 | Auto-apply New Clause/Schedule amendments to the same location in order of their Sort Code value | Pass |
LDAPP-4467 | LMS-167 | Accept applied amendments one-by-one | Pass |
LDAPP-4473 |
| Ignore the insertion of new content by other amendments when determining the start and end locations of other auto-applied amendments | Pass |
LDAPP-4515 | LMS-2742 | Accept auto-applied amendments which insert a Cross-heading, Part, or other grouping level in the Body or Schedules of a Bill | Pass |
LDAPP-4543 | LMS-3250 | Auto-apply amendments which leave out a string beginning with punctuation | Pass |
LDAPP-3107 | LMS-2742 | Auto-apply simple textual amendments to Tables in a bill version | Pass |
LDAPP-4480 |
| Improve Editor performance in documents containing lots of inline amendments | Pass |
LDAPP-4510 |
| Remove beta label from inline amendment option in Create LoDA modal | Pass |
LDAPP-4523 |
| Auto-apply amendments which insert new Parts/Chapters/cross-headings Before provisions in the Body or Schedules of a Bill | Pass |
SPT-1750 | PRB1001014 | Content is duplicated when accepting an amendment | Pass |
SPT-1742 | PRB1001014 | Incorrect application of amendments applied causes bulk amendment application to fail | Pass |
Amendment lists
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4384 | LMS-2691 | Commons: small changes to the template text which appears in various Official Lists | Pass |
SPT-1753 | PRB1001044 | Interstitial Heading not formatting to italics | Pass |
PDF generation
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4504 |
| Add hyphenation options to tables | Pass |
LDAPP-4511 |
| Fix line numbering issues caused by hyphenation | Pass |
SPT-1688 |
| Fix rendering issues caused by the interaction between tracked changes, comments, line-numbering, and running header in Scottish bills | Pass |
Improved sorting of amendments on the Amendment Manager tab
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4519 |
| Enable accept/reject/reply actions on tracked changes which affect content inside locked elements | Pass |
LDAPP-4508 |
| Update Adjacent quoted structure behaviour to remove restrictions | Pass; but note one false-positive Document Check error which is now appearing in certain places where Adjacent Quoted Structures are used |
LDAPP-4481 |
| Insert Quoted Structure from CCA | Pass |
LDAPP-4544 |
| Fix the quick-fix links in the validation tooltips which appear in the Editor | Pass |
LDAPP-4477 |
| Update schematron rules for v.18 | Pass |
LDAPP-4463 |
| Remaining problems of inline styling | Pass |
LDAPP-4547 |
| Fix issues with cursor placement and scroll position | Pass; there may be some scenarios where the scroll position changes unexpectedly but the majority of cases are greatly improved. |
LDAPP-4548 |
| Create a separate right-hand panel to show schema errors | Pass |
SIs/SSIs
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4541 |
| Add chapter number to primary legislation document references in citation footnotes | Pass |
LDAPP-4522 |
| Make table number bold in PDFs | Pass; note that this also works correctly in bills |
LDAPP-4521 |
| Change order of SI/SSI procedures in drop down when creating new project | Pass |
LDAPP-4545 | LMS-3256 | Insert italic heading in SI/SSI schedules | Pass |
LDAPP-4562 |
| Increase space before the signature block in SIs/SSIs | Pass |
Document Management and Permissions
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LNI-8 |
| Update Significant Version labels when a bill is published | Pass |
LNI-39 |
| Update labelling of document versions (PDFs, save snapshot & published bills) | Pass |
SPT-1723 |
| Applying a permission to a new doc in a project hides the project from everyone else | Pass |
References
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4500 |
| Improve handing of contextual references to include "after", "before" and "omit" | Pass |
LDAPP-4499 |
| Improve handling of alias references: context and brackets | Pass |
LDAPP-4479 |
| Incorrect tagging of word as external reference ('that partial') | Pass |
LDAPP-4447 |
| Prevent //ref tags from appearing inside //def tags when tagging references | Pass |
LDAPP-4428 |
| Fix duplicate start quote when tagging document alias definitions containing inline elements | Pass |
LDAPP-4532 | LMS-3255 | Handle list of references of different types | Pass |
SPT-1754 | PRB1001026 | Internal refs are tagged with the wrong context | Pass |
SPT-1752 | PRB1001042 | X-ref tags placed in the wrong location when an Act title is incomplete or preceded by the word "into" | Pass |
SPT-1751 | PRB1001049 | Tags added in the wrong places when using the 'Tag x-refs' feature on a version of a bill which contains line numbering | Pass |
Origins and Destinations
Development ticket | Feature request or support ticket reference | Description | Test result |
|---|---|---|---|
LDAPP-4520 |
| Change approach to Origin notes to use //authorialNote | Pass |
LDAPP-4535 |
| Automatically tag external references which appear in Origin notes | Pass |
LDAPP-4325 |
| Prototype generate of table of origins | Pass |
LDAPP-4551 |
| Prototype generation of Table of Destinations | Pass |
LDAPP-4528 |
| Expand/collapse all origin notes in the Editor | Pass |
LDAPP-4529 |
| Display Origin information in a generated PDF | Pass |
Annex - Overview of testing and development processes
This annex outlines how we implement quality control in relation to changes to the Lawmaker application during the development process and as part of the release process.
Testing during the development process
All new features and bug fixes are implemented by a multi-functional development team operating in an agile manner.
To ensure the quality of those changes, we implement the following steps:
The initial developer working on a feature or bug fix will, at the same time as writing the necessary code, write automated tests (“unit tests”) or modify existing ones, that can be run in future to ensure the feature/fix is still working as intended.
A second developer peer reviews the codes changes made by the first developer and the tests, before they are merged into the development environment.
When a change is merged into the development environment, all existing unit tests are run against the updated version to ensure that they still pass. If they don’t then the change will not be merged.
The new feature or fix is then tested by a QA engineer or another developer in the development environment to ensure it works as expected. A member of the Lawmaker service team tests feature/fix to ensure it meets the original specifications/expectations (this is effectively “user acceptance” of the change within the agile cycle).
If an issue is found at any of these stages, either it is passed back to an earlier stage in the process to be resolved or a separate task is created to resolve the issue.
Testing as part of the release process
Before any significant or minor release, a series of additional testing is carried out to ensure the quality of the release. The aim of this testing is to ensure that all new features and fixes work as expected when integrated together and also that no regressions in existing functionality have been introduced.
It is impossible to completely rule out any possibility of a regression being introduced somewhere but the testing aims to ensure that all business critical paths through the application work as expected and any defect introduced has only a minor impact on users.
So far as possible all issues found during release testing will be rectified before the release but work is prioritised as follows:
if an issue affects a business critical paths then it will be fixed and retested before release,
if it otherwise has a non-minor impact on users or could significantly erode user confidence in the robustness of the Lawmaker service then they will be fixed and retested before release,
if it has a minor impact on users or is a pre-existing issue that is not being introduced by the release then it may be recorded in our issue-tracking system and addressed in a later release.
As part of the release process, the following steps are taken to ensure the quality of the release
An updated version of Lawmaker incorporating all changes is deployed to the Staging environment, which (apart from scale) is an exact replica of the Production environment.
Each new feature and bug fix included in the release is re-tested in the Staging environment.
A series of “end-to-end” test scripts are run through to ensure that the business-critical paths through the application work as expected. These scripts cover, in particular:
Drafting and managing a Scottish Bill
Drafting and managing amendments and the amendment process for a Scottish Bill
Drafting and managing a UK Bill
Drafting and managing amendments and the amendment process for a UK Bill
The “ping pong” process for UK Bills
Drafting and submitting an SI/SSI
A series of specific tests on particular areas of Lawmaker functionality are carried out, including:
individual features and functions within the Editor,
PDF generation,
converting inline amendments to traditional amendments, and
automatically applying amendments to a Bill.
Additional ad hoc testing by the Lawmaker Service Team also takes place in relation to any aspect of the application that is considered to be at particular risk of regression as a result of the changes made.
A series of automated load tests are carried out to ensure that the changes made in the new release do not have an impact on the robustness of the application.
Senior users within each user group are notified when the updated version of Lawmaker is deployed to the Staging environment and given the opportunity to view the changes and carry out ad hoc testing if they have capacity. Any feedback received from those users is reviewed and may either feed into further changes before the release (e.g. if a bug is identified) or will be recorded so that it can be addressed in future releases.