(For context, see Overview of testing and development processes.)
Summary of v.17.1 testing
Release note
Significant issues found during testing
The following issues were identified during testing and resolved:
The following issues were identified during testing but not resolved:
Testing of bug fixes
https://lawmaker.atlassian.net/wiki/spaces/RTS/database/571211778?savedViewId=59dc37bc-dbbb-437f-b9cc-693d2d7ec900Regression tests performed
End-to-end tests
The following end-to-end regression scripts and specific feature scripts were run through by the Lawmaker team:
Script | Test result | Notes |
|---|---|---|
UK Bill Test Script | Pass | |
UK Amendments Test Script - | Pass | |
UK Ping Pong test script | Pass | |
Editor Test Matrix script | Pass | Issue identified with the display of images in the Editor. Fixed in the final candidate build. |
Scottish Bill test script | Pass | |
Scottish Amendments test script | Pass |
Additional UK Bill and Amendment testing
In addition to the general regression scripts, additional testing was carried out of UK Bill functionality using real data from previous sessions. This focused particularly on the auto-apply amendment functionality and ensuring that no regressions had been introduced by comparing the output of the auto-apply feature in the new version of Lawmaker with the output from the current version of Lawmaker.
Bill title | Amendments tested | Notes |
|---|---|---|
Renters Reform (HoC, As Introduced) | 170 OPC amendments | Auto-apply outputs from Production compared with Staging; no differences found |
Digital Markets, Competition and Consumers (HoL, As Brought from the Commons) | 50 amendments tested | Auto-apply outputs from Production compared with Staging; no differences found |
Leasehold and Freehold Reform (HoC, As Introduced) | 124 amendments tested | Auto-apply outputs from Production compared with Staging; no differences found |
PDF testing
PDFs were generated for a range of real and test documents and the outputs compared to PDFs generated with the current version of Lawmaker to identify and issues.
No issues were found.
New and improved feature testing
Bug fix testing
The following bug fixes in version 17.1 were tested
SI/SSIs
Issue key | Summary | Test result |
|---|---|---|
SPT-1703 | Error when copying an SI to another project when SI doesn't have enacting text | Pass |
SPT-1700 | Smart paste of SI prov1 with auto-renumber ON strips trailing full stop from num | Pass |
SPT-1695 | Updated SI subject heading list | Pass |
SPT-1689 | SI Paragraph (1) spacing differs in quoted structure as starting element | Pass |
SPT-1679 | Specific wordings of SI/SSI dates still causes line break | Pass |
SPT-1676 | Procedure of SI on the Project tab or Dashboard is locked in after creation and unchangeable | Pass |
SPT-1671 | Add OFGEM and OFCOM as new organisations | Pass |
Editor
Issue key | Summary | Test result |
|---|---|---|
SPT-1713 | Editor fails to revert to using the correct document filter after an unknown action in the editor results in a failure to switch back to the LDAPPDocumentFilter | Pass |
SPT-1690 | Wrong placement of material dragged into containers in amendment QSs | Pass |
SPT-1711 | Oxygen sometimes fails to retrieve the XML from the server for operations like auto-save, save and structure view updates | Pass |
SPT-1706 | The "A problem has occurred" modal appears in documents with lots of spelling errors. | Pass |
SPT-1702 | Performance issue in getElementByGuid(Oxygen client-side code) (included in patch release 16.2.1) | Pass |
SPT-1701 | TNA - Screen Jumps On Paste | Pass |
SPT-1699 | Poor performance when copying a reasonable number of provisions (included in patch release 16.2.1) | Pass |
SPT-1683 | Mark as done toggle not working in certain cases | Pass |
SPT-1672 | Issues with rejecting "surround" tracked changes in Oxygen v25.1. | Pass |
SPT-1653 | Updates to document during a renumber operation result in strange document modifications | Pass |
SPT-1656 | Promote/Demote of closing words unwraps inline formatting and footnotes | Pass |
SPT-1631 | Incorrect Document Push in Copy Provisions Operation | Pass |
SPT-1616 | Strange restrictions on using the Structure View to move a Schedule in a version of a bill where a New Schedule amendment has been applied | Pass |
SPT-1611 | Oxygen exceptions occurring when a user accepts tracked change edits ("A problem has occurred") (included in patch release 16.2.1) | Pass |
SPT-1609 | Cross references inside tracked change deletions are re-added without deleted markup after updating cross references in the provision | Pass |
PDFs
Issue key | Summary | Test result |
|---|---|---|
SPT-1696 | Quoted structure in change track have incorrect indent | Pass |
SPT-1693 | Text in ukpga quoted structure //section/intro is not justified in PDF | Pass |
SPT-1692 | Page numbering broken from page 10 onwards in Scottish Parliamentary Amendment lists | Pass |
SPT-1691 | Empty paragraph element wrongly aligned in PDF | Pass |
SPT-1682 | Long hyperlinks (URLs) that are split across multiple lines have line spacing issues in the PDF and CSS issues in footnotes in the editor | Pass |
SPT-1558 | Resolve PDF rendering issues caused by missing attributes on quotedStructures (ukl:docName and ukl:indent) | Pass |
Amendments
Issue key | Summary | Test result |
|---|---|---|
SPT-1651 | Auto-apply: amendments which leave out words after a Defined term can't be applied if the quotation marks on the defined term are included in the amendment instruction | Pass |
SPT-1646 | Extra spaces appear in certain substituting amendments when auto-applied to a bill version | Pass |
SPT-1633 | Auto-applying amendments to rref elements sometimes results in too much text being included in the rref; original rref's href attributes also persist | Pass |
SPT-1632 | Auto-applied amendment which inserts text beginning with punctuation after a <ref> element adds the text inside the <ref> when accepted | Pass |
SPT-1266 | Side-lining on auto-applied insert section amendment goes down too far | Pass |
SPT-1608 | Moving an amendment in a LoDA using the Structure View with tracked changes turned on results in the amendment being duplicated | Pass |
SPT-1541 | Auto-saved traditional amendments in an inline LoDA cannot be restored into the working version | Pass |
Document management (Project tab, comparison, permissions etc.)
Issue key | Summary | Test result |
|---|---|---|
SPT-1662 | Duplicating an 'Auto backup' snapshot results in the wrong XML content being retrieved - later changes are included in the duplicate | Pass |
SPT-1218 | User who first created document still has read-only access when their permissions are removed | Pass |
SPT-1677 | ALL CAPS text is displayed with additional spacing in the Edit Portion dropdown | Pass |
SPT-1648 | Opening an Edit portion document in parallel with others causes issues | Pass |
SPT-1645 | When logging out with two Editor tabs open (one containing a portion of a version), it's not possible to 'Save all changes' in the 'Unsaved changes' modal | Pass |
SPT-1665 | Comparison output moves opening words into the wrong position on Scottish Bill | Pass |
SPT-1657 | XML comparison functionality doesn't handle processing instructions appropriately | Pass |
SPT-1627 | Incorrect comparison result when an x-ref has been updated within a <mod> | Pass |
Annex - Overview of testing and development processes
This annex outlines how we implement quality control in relation to changes to the Lawmaker application during the development process and as part of the release process.
Testing during the development process
All new features and bug fixes are implemented by a multi-functional development team operating in an agile manner.
To ensure the quality of those changes, we implement the following steps:
The initial developer working on a feature or bug fix will, at the same time as writing the necessary code, write automated tests (“unit tests”) or modify existing ones, that can be run in future to ensure the feature/fix is still working as intended.
A second developer peer reviews the codes changes made by the first developer and the tests, before they are merged into the development environment.
When a change is merged into the development environment, all existing unit tests are run against the updated version to ensure that they still pass. If they don’t then the change will not be merged.
The new feature or fix is then tested by a QA engineer or another developer in the development environment to ensure it works as expected. A member of the Lawmaker service team tests feature/fix to ensure it meets the original specifications/expectations (this is effectively “user acceptance” of the change within the agile cycle).
If an issue is found at any of these stages, either it is passed back to an earlier stage in the process to be resolved or a separate task is created to resolve the issue.
Testing as part of the release process
Before any significant or minor release, a series of additional testing is carried out to ensure the quality of the release. The aim of this testing is to ensure that all new features and fixes work as expected when integrated together and also that no regressions in existing functionality have been introduced.
It is impossible to completely rule out any possibility of a regression being introduced somewhere but the testing aims to ensure that all business critical paths through the application work as expected and any defect introduced has only a minor impact on users.
So far as possible all issues found during release testing will be rectified before the release but work is prioritised as follows:
if an issue affects a business critical paths then it will be fixed and retested before release,
if it otherwise has a non-minor impact on users or could significantly erode user confidence in the robustness of the Lawmaker service then they will be fixed and retested before release,
if it has a minor impact on users or is a pre-existing issue that is not being introduced by the release then it may be recorded in our issue-tracking system and addressed in a later release.
As part of the release process, the following steps are taken to ensure the quality of the release
An updated version of Lawmaker incorporating all changes is deployed to the Staging environment, which (apart from scale) is an exact replica of the Production environment.
Each new feature and bug fix included in the release is re-tested in the Staging environment.
A series of “end-to-end” test scripts are run through to ensure that the business-critical paths through the application work as expected. These scripts cover, in particular:
Drafting and managing a Scottish Bill
Drafting and managing amendments and the amendment process for a Scottish Bill
Drafting and managing a UK Bill
Drafting and managing amendments and the amendment process for a UK Bill
The “ping pong” process for UK Bills
Drafting and submitting an SI/SSI
A series of specific tests on particular areas of Lawmaker functionality are carried out, including:
individual features and functions within the Editor,
PDF generation,
converting inline amendments to traditional amendments, and
automatically applying amendments to a Bill.
Additional ad hoc testing by the Lawmaker Service Team also takes place in relation to any aspect of the application that is considered to be at particular risk of regression as a result of the changes made.
A series of automated load tests are carried out to ensure that the changes made in the new release do not have an impact on the robustness of the application.
Senior users within each user group are notified when the updated version of Lawmaker is deployed to the Staging environment and given the opportunity to view the changes and carry out ad hoc testing if they have capacity. Any feedback received from those users is reviewed and may either feed into further changes before the release (e.g. if a bug is identified) or will be recorded so that it can be addressed in future releases.