This essay chronicles a short dive in to some of the evidence supporting New Zealand’s Smokefree Aotearoa 2025 policy. I first became aware of this initiative through coverage on BBC News . I am currently undertaking a PhD in policy modelling and simulation, so I am interested in understanding what evidence (especially modelling evidence) has been used by the New Zealand government to support these proposals. In particular, the government is proposing to introduce a prohibition on tobacco sales to anyone born after a certain date. From the BBC News article:
New Zealand will ban the sale of tobacco to its next generation, in a bid to eventually phase out smoking. Anyone born after 2008 will not be able to buy cigarettes or tobacco products in their lifetime, under a law expected to be enacted next year.
Prohibition of any substance is controversial and has a history of not being 100% effective. However, my objective in this essay is not to comment on the proposal itself, but rather to describe the process I have gone through to start to understand the modelling evidence in support of this proposal. In particular I am concerned with the democratic transparency of this evidence, by which I mean the ease to which citizens are able to access, interpret, and evaluate this evidence, and thus determine for themselves if the evidence is sufficient, and if the policy is sound. I will take this investigation step-by-step and comment on the ‘speed-bumps’ – factors which reduce democratic transparency – that I discover along the way, along with recommendations to alleviate these problems, in italics.
The first step was to conduct a Web search, since the BBC News article doesn’t link to any other pages relevant to these policy proposals. The Guardian also covered this story and likewise did not link out to any sources at the New Zealand government. This is the first speed-bump on the policy investigation journey: News outlets regularly fail to link to primary evidence. This is not always the case, as demonstrated in another recent BBC News article, however in my personal experience it is quite common. The problem with not linking out is that it increases the time it will take interested readers to find the relevant primary material. Further, it will discourage some readers from digging deeper at all: there is a greater amount of ‘activation energy’ required to formulate search terms, type them in to a search engine, and dig through the results looking for that initial foothold, compared to simply clicking a link. Therefore my first recommendation is: News outlets should always link to primary sources, such as journal articles, policy briefs, etc. If you are a journalist, add a link in your article. If you are interviewed for an article about work you have produced, insist on a link to that work in the article. Having a link makes it very easy for people, especially those with limited research skills, to start engaging more closely with the evidence.
I decided to conduct a Web search for “new zealand tobacco ban”. The first page of search results consisted entirely of news coverage from other outlets, but on the second page I found a link to a page on the New Zealand Ministry of Health , which I then followed because that is most likely the origin of the policy recommendations. From here it is easy to find some resources such as the action plan , however this document does not contain references to any evidence in favour of the proposal to create a smokefree generation. In fact there is only one reference to modelling in the entire document:
The development of New Zealand’s tobacco control programme over many years has been closely modelled on the FCTC. New Zealand remains committed to supporting the implementation of the FCTC globally.
and this statement provides no citation to support it! There are also multiple references to evidence and evidence-based approaches throughout the document but again, most of these provide no citation or further information. This is the second speed-bump: Policy publications,particuarly those aimed at the general public, may not provide adequate references to the supporting evidence. Again this is not always the case, but in my experience of going on these policy-evidence-investigation journeys, it happens surprisingly frequently. This is much more problematic than the previous issue of news outlets not linking to primary sources, because often there are multiple sources of evidence which have been considered to inform these policy documents, and it is much more difficult to determine the relevant primary sources from a nebulous mention of ‘evidence’ with no additional information. Thus my second recommendation, to authors of policy publications: Always include ample citations in your policy publications, regardless of the intended audience, and especially for any sentences mentioning ‘evidence’, ‘data’, or ‘models’.
Returning to the web pages for Smokefree Aotearoa 2025, I found a reference to another document, “Proposals for a Smokefree Aotearoa 2025 Action Plan”, although this was not hyperlinked. I was able to find this through a web search, and it can also be found via the ‘Publications’ page on the Ministry’s website, but this is another speed-bump: Websites are not always sufficiently hyperlinked. The solution is very simple: If you are making reference to another document on your website, hyperlink your reference.
After opening the “Proposals for a Smokefree Aotearoa 2025 Action Plan” consultation document I am able to find three instances of the word ‘evidence’ and four instances of words matching ‘model’ (e.g. ‘modelling’, ‘models’), the majority of which have proper citations. Now we are able to start diving in to the evidence proper. There is one particular reference to the smokefree generation, which we will investigate further:
New Zealand modelling suggests that, if well enforced, a smokefree generation policy would halve smoking rates within 10 to 15 years of implementation. The health gains per person would be five times larger for Māori than for non-Māori (Blakely et al 2018).
The reference is hyperlinked and takes us to “Modelling the number of quitters needed to achieve New Zealand’s Smokefree 2025 goal for Māori and non-Māori” from the New Zealand Medical Journal. The methods section states:
We used the established BODE3 tobacco forecasting model1–3,6,7 to project smoking prevalence separately for Māori and non-Māori to 2025 under a business-as-usual (BAU) scenario.
but does not provide any detail on this model within the article itself. This is slightly unusual to me, especially for a journal with such a large mandate, as much of the readership may not be familiar with tobacco forecast modelling. I would expect the article to include at least a brief description of how this model works, and more importantly, what the underlying assumptions are. Understanding the mechanisms and assumptions is vital for at least two reasons. First, human brains are very good at making assumptions and ‘filling in the blanks’, and unless we are told what a model is doing ‘under the hood’, we will make assumptions about the mechanisms and validity of the model; without a description of the model we cannot know how it differs from our own mental model, or how well the model aligns with other sources of evidence. Second, all models have a range of inputs and parameters under which we can expect them to behave reasonably and provide accurate results, and outside of this range the confidence in their outputs fall off, sometimes quite dramatically. For example, the Newtonian and Einsteinian models of gravity break down at subatomic scales. Understanding the mechanisms and assumptions inherent in a model allow us to start to delineate the regions of confidence of the model, and decide how much confidence we can put in the results. This is the next speed-bump: It is treacherous to interpret model results without an adequate understanding of the assumptions inherent in the model. Thus I recommend to researchers that: When writing the methods section of papers that make use of any models, briefly describe the mechanisms behind the models used, and clearly state the assumptions which have been made as part of the model’s development.
Following one of the references takes us to “What will it take to get to under 5% smoking prevalence by 2025? Modelling in a country with a smokefree goal” by Ikeda et al. in the journal Tobacco Control, and fortunately this paper is open access and available for free. Although we have been lucky to avoid the closed access speed-bump this time, it is still common to encounter journal articles which are only accessible for a fee, and the issue should be stated: Closed access publications prevent members of the public, and any other individuals who are unaffiliated with a subscribing institution (which may include policy-makers), from engaging with research, due to exorbitant access fees. The open access movement has already made a huge impact and momentum is continuing to build, but policy still continues to rely on closed access publications. From the perspective of democratic transparency, the solution is clear: All research and publications used to inform policy recommendations must be made accessible to the public for free. If you are a researcher, consider publishing in an open access journal, and always self-archive your work on your personal website, in your institution’s archive, and/or through preprint services such as arXiv. If you are authoring policy literature which references closed access research, attempt to locate open access copies to ensure they are available, and if open access copies are not available, contact the authors to request a copy which you can freely archive and distribute, and then link to it from your own literature. One way governments and third sector organisations can facilitate free access to research is through establishing their own institutional open access archives, into which they can deposit published, postprint (author-accepted manuscript), or preprint copies of all evidence used, which can then be directly linked to from policy documents. Establishing such archives would also confer additional benefits to the public, the evidence-consuming organisations, the researchers producing the evidence, and the open access movement.
Let’s return to Ikeda’s paper. This paper is not the original source of the model, but it does give an overview of how the model works:
Unless stated otherwise, the modelling approach we used is as described in a published Australian model by Gartner et al.8 Here, we briefly overview the key features of that model, and emphasise the adaptations for this modelling study and the New Zealand input data (eg, ethnic group specific modelling).
The paper by Gartner is also open access and itself references another paper by Mendez, this time in the American Journal of Epidemiology, which again is thankfully open access, and this traces the complete lineage of the model. The original article by Mendez provides the mathematical specification from which the other models will have been constructed.
Returning again to the paper by Ikeda, it is revealed that the model has been implemented in Microsoft Excel, which presents some advantages and some disadvantages. On the one hand, it is possible to develop a robust model in Excel; and it is widely available software with which many people are familiar, and thus it is provides a good baseline of democratic transparency. On the other hand, it is proprietary software (albeit with some largely-compatible open source alternatives), which presents financial and technical barriers for some users; its file format is not amenable to version control with popular tools such as git (although there are other mechanisms built in to Excel such as change tracking and co-authoring); and it lacks built-in tooling for test automation, debugging, and other features which can be used to improve and maintain reliability and confidence (although there are libraries like xlwings which allow interfacing with spreadsheets from conventional programming languages, and which can be used to develop such tooling). Therefore, using Excel to build a model is neither good nor bad, but it does represent a particular set of trade-offs. An alternative option is to write models in conventional open source programming languages such as R and Python, particularly in a literate programming style, and as programming literacy increases over the years the democratic transparency gap between Excel models and non-Excel models may start to shrink to the point that the benefits of modelling outside of Excel begin to tip the scales. The potential speed-bump here is: Excel is not always a suitable tool for modelling, particularly as models become more complex, and when models are developed by multiple authors. In this case, the model is not too complicated, but it does make use of a separate add-in for Monte Carlo simulation, and so this model may be approaching the limit of what is appropriate. The recommendation here: Model builders must understand their requirements, and the potential complexity of their model, before they start coding. Model builders must evaluate the available tools that they could use to implement their model; take stock of the trade-offs inherent in these tools; and choose the right tool for the job given these trade-offs, the requirements, and the model’s innate complexity.
Unfortunately, although the model was built in a spreadsheet, it doesn’t seem that the spreadsheet has been published, as it is not included in the supplementary materials links at the bottom of the article. This is a speed-bump which, like open access, the research community has started to become more aware of: Papers are only one output of the research process. Code and data which have been used to produce research often goes unpublished. Some datasets are sensitive and contain information which cannot be made publicly available for safeguarding and anonymity reasons. However, other datasets can be sufficiently anonymised (with varying amounts of effort), and code is almost always safe to publish. Not publishing data and code presents a number of problems. First, it makes it harder for other people to investigate and replicate the original study. This is important because there may be issues in the data or the processing of the data which affect the study’s conclusions: data may not be representative; data may not have been pre-processed and cleaned in an appropriate way; model code may contain bugs; stochastic models may be sensitive to choices of seeds for random number generators; and models outputs may be sensitive to choices of particular parameters. The methods section of the paper serves to document the pipeline through which the data has flowed, but it is only a description, and the ground truth is the code itself; the map is not the territory. Second, not publishing code and data makes it harder for other people to conduct additional analyses and build on the existing work, which slows down the scientific process. Often you can email the corresponding author of a paper and they will be happy to provide you with code and any non-sensitive data they have used, but most researchers are busy people, and it may take days or weeks for them to respond to your query, and there is no guarantee that you will get a reply. My recommendation is: Research papers must link to all code and non-sensitive data that has been used to produce the results they contain. This includes any code used for data pre-processing, and data presentation, not just model code. Any sensitive data which cannot be published should be documented in a manifest and this manifest should be published along with the other materials. If you are a researcher, archive your supplementary materials on OSF.io, Github, or another public archive, link to this material in your paper, and submit all material to the journal as supplements to be made available with your paper online when it is published. If you are referencing the research, ensure all materials are readily available, and if you cannot find them, contact the corresponding author of the paper and ask them to either publicly archive the materials at one of the previously mentioned archives, or to provide you with a freely distributable copy of all materials which you can then go and archive yourself. When you are referencing the paper in your own work, link to this material alongside the reference.
It is at this point that I will draw this journey to a close. As I have demonstrated, digging in to a policy brief and finding the evidence to support its recommendations is a long process, with many steps along the way. Barriers can present themselves at each step, and there are many small improvements which can be made by researchers and research consumers which together add up to a big win for democratic transparency. It is important for everyone involved in the policy-making process to take a holistic view of how the public can engage with evidence, and to eliminate speed-bumps wherever possible.
References