blob: f5e5f8e066d4d0381daab3680a68618549ee3c8c [file] [log] [blame]
Govindraj Rajaae67fb42024-12-12 17:16:03 -06001LTS - Long-Term Support
2=======================
3
4.. table:: Table 1: Document History
5
6 +-------------+--------------------+-------------------------------------------------------+
7 | Date | Author | Description |
8 +=============+====================+=======================================================+
9 | 2022-07-20 | Okash Khawaja, | Initial draft. |
10 | | Varun Wadekar | |
11 +-------------+--------------------+-------------------------------------------------------+
12 | 2022-07-21 | Varun Wadekar | Refine the Maintainership guidelines and planning |
13 | | | sections. Introduce a new section documenting a day |
14 | | | in the life of a LTS branch maintainer |
15 +-------------+--------------------+-------------------------------------------------------+
16 | 2022-08-05 | Okash Khawaja, | Merge two drafts (draft 1 and 2), address comments |
17 | | Varun Wadekar | made by both authors, cosmetic changes to the content |
18 | | | all over the document |
19 +-------------+--------------------+-------------------------------------------------------+
20 | 2022-08-05 | Okash Khawaja | Add note about testing support available from TF.org |
21 +-------------+--------------------+-------------------------------------------------------+
22 | 2022-08-05 | Varun Wadekar | Changed the “Future plans” section to “FAQ” and |
23 | | | answered some of the questions with feedback from |
24 | | | the community. |
25 +-------------+--------------------+-------------------------------------------------------+
Govindraj Raja8f88d032025-01-08 18:44:38 -060026 | 2025-01-07 | Govindraj Raja | Convert from pdf to rst. |
27 +-------------+--------------------+-------------------------------------------------------+
28 | 2025-01-07 | Govindraj Raja | Updates based on learnings and suggestions. |
Govindraj Rajaae67fb42024-12-12 17:16:03 -060029 +-------------+--------------------+-------------------------------------------------------+
30
31This document proposes a plan for long-term support (LTS) of the |TF-A| project.
32
33Why is LTS required?
34--------------------
35LTS is needed for commercial reasons. More specifically, on the device side,
36when a product is released, the companies have to support that in-market product
37such that the amount of changes to the firmware are kept to a minimum to avoid
38the risk of regression. At the same time the companies don't want to exclude
39critical patches such as those for security advisories. Similarly on the server side,
40companies want to minimize the churn when deploying fixes during incident
41response, e.g. due to critical security bugs.
42
Govindraj Raja8f88d032025-01-08 18:44:38 -060043Also, the European Cyber Resilience Act (CRA) is a new EU legislation that mandates
44cybersecurity standards for products containing digital elements, aiming to
45protect consumers and businesses by ensuring manufacturers build security into
46their hardware and software throughout their lifecycle, including automatic
47updates and incident reporting; essentially requiring all digital products
48sold in the EU to meet specific cybersecurity requirements.
49
50This means that companies have to maintain and backport critical updates to
Govindraj Rajaae67fb42024-12-12 17:16:03 -060051old branches internally. As this effort is duplicated across different companies
52using TF-A, it makes sense to factor out this effort into a community-wide LTS.
53
54What does LTS mean for TF-A?
55----------------------------
56In this section we will define exactly what constitutes LTS for TF-A.
57Specifically, we will define the following characteristics:
58
59- criteria for selecting patches which will be backported to LTS branches
60- lifetime and frequency of LTS branches
61
62**Criteria**
63
64We must have an objective criterion for selecting patches to be backported to
65LTS branches. This will make maintenance easy because:
66
67a. there will be less -- ideally no -- discussion when selecting patches to backport
68b. large parts of the process can be automated
69
70Below is the criteria
71
72#. No features will be backported.
73#. Security advisories: Any patch that makes it into :ref:`Security Advisories`
74 is automatically selected for back porting. This includes patches to external
75 components too, e.g. libfdt.
76#. Workarounds for CPU and other ARM IP errata
77#. Workarounds for non-ARM IP errata, e.g. TI UART
78#. Fixes for platform bugs. These patches must not modify any code outside of
79 the specific platform that the fix applies to.
80#. Patches can only be backported from the master branch. In other words, the
81 master branch will be a superset of all the changes in any LTS branch.
82
83**Lifetime and frequency**
84
85This section approaches three questions: for how long should an LTS release be
86supported, how frequently should LTS releases be made and at which time(s) of
87the year should the releases be made.
88
891. For how long should an LTS release be supported?
90
Govindraj Raja8f88d032025-01-08 18:44:38 -060091The Linux kernel maintainers supports an LTS branch for 2 years. Since firmware
92tends to have less churn and longer lifetime than a HLOS, TF-A is trying to
93support at-least 7 years for its LTS. Initially it was intended to support
945 years but there has been no objections to extend LTS support to 7 years.
95There are many challenges that may influence the 7 year support from CI
96infrastructure to availability of maintainers.
Govindraj Rajaae67fb42024-12-12 17:16:03 -060097
982. How frequently should LTS releases be made?
99
100Given that many products that have a release cycle, have a yearly release
101cycle, it would make sense to have yearly TF-A releases.
102
1033. Which time(s) of the year should the releases be made?
104
105TF-A releases are cut twice a year: May and November. Basing LTS release
106on the November TF-A release has a few benefits. First, it aligns with Linux
107LTS releases which happen towards the end of each year. Second, it aligns
108with Android releases which tend to fall in Q3 each year. Since product
109releases are timed with Android release, this gives enough time to harden
110the TF-A LTS release during development so that it's ready for launch in
111Q3 following year. On the other hand, if the May release of TF-A is chosen as
112the basis for LTS then developers will have little time -- about a month,
113taking into account the test-and-debug phase before LTS is cut (see below) --
114before Android release.
115
116To summarize, there will be one LTS release per year. It will be supported for
1175 years and we can discuss extending it to 7 years later on. The LTS release
118will be based on the November release of TF-A.
119
120**Testing Criteria**
121
122Every patch merged to the LTS branch will complete the following tests before
123getting approved.
124
125#. TFTF tests currently running in the testing farm
126#. CI/CD static analysis scans
127#. Coverity scans
128#. Platform tests
129
130Platforms that are not maintained upstream will undergo testing downstream in a
131pre-defined window. The platform maintainer will complete the testing and provide
132a verified score on the patch once testing is completed.
133
134** A note about test coverage from TF.org **
135
136Currently TF.org maintains a CI system to run TF-A automated tests on a
137selection of HW boards donated by TF.org members (a benefit reserved to project
138members, see the project charter for more details). This automated test coverage
139will be extended to cover testing for LTS as well for boards that are part of
140the CI system.
141
Govindraj Raja8f88d032025-01-08 18:44:38 -0600142**TFTF Branching**
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600143
144A note about testing here. After a patch is backported to an LTS branch, that
145branch will need to be regression tested. Since TFTF moves forward with latest
146TF-A changes, newer TFTF tests may not apply to old LTS branches. Therefore
147TFTF will also need to be branched, in-sync with TF-A LTS branches. In other
148words, there will be one TFTF LTS branch corresponding to each TF-A LTS branch.
149The TFTF LTS branch will be used to regression test the corresponding TF-A LTS
150branch.
151
152As we work with the LTS branch of TFTF, we might also need fixes for TFTF
153itself to be ported to LTS. However, decision-making about those patches need
154not be as stringent as for TF-A.
155
Govindraj Raja8f88d032025-01-08 18:44:38 -0600156**CI Scripts**
157
158CI Scripts moves forward with TF-A changes, since we need to checkout the
159corresponding release version of CI scripts for LTS.
160
161Though we are unlikely to update CI scripts, but time to time migrating a newer
162FVP version or deprecating certain tests due to unavailability of platforms may
163influence updates to CI Scripts.
164
165**Hafnium / OP-TEE**
166
167Both Hafnium and OP-TEE move forward with TF-A changes so we need to freeze their
168corresponding version from TF-A release for a LTS.
169
170**MbedTLS**
171
172Updates to the version of MbedTLS used with LTS will happen time to time based on
173maintainers call to update them or not.
174
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600175Release details
176---------------
177This section goes into details of what the LTS release process will look like.
178
179
180**Test-and-debug period**
181
182Since the LTS branch will be used in product releases, it is expected that more
183testing and debugging will be done on the November release of TF-A. Therefore
184it would make sense to leave at least a month after the November release and
185then cut the LTS branch. We recommend two months, given that one of the months
186is December which tends to be slower due to holidays. So, an end-of-November
187TF-A release would result in a beginning-of-February LTS release. Note that
188the LTS branch will be created at the same time as the TF-A November release,
189but it will be officially released at the end of January or early February.
190Going forward we should strive to make the period smaller and smaller until
191ideally it coincides with TF-A November release which means that our test
192and CI/CD infra is good enough to allow that to happen.
193
194**Example timeline**
195
196Below is an example timeline starting from the November 2022 release of TF-A.
197
198.. image:: ../resources/diagrams/lts-timeline-example.png
199
200- Nov 2022: TF-A 2.8 is released towards the end of Nov, 2022. Not shown in the
201 diagram, at the same time LTS release candidate branch is made which is based
202 on TF-A 2.8. This means new features going in 2.8 won’t go in the LTS branch.
203 We can call it `LTS 2.8-rc`.
204- Feb 2023: After testing and debugging LTS 2.8-rc for a couple of months,
205 LTS 2.8.0 is officially released in early Feb 2023.
206- May 2023: TF-A 2.9 is released but since this is not an LTS branch it doesn’t
207 affect LTS.
208- Somewhere between May and Nov of 2023: A security advisory comes up and the
209 related patches go into TF-A master branch. Since these patches fall under
210 LTS criteria, they are backported to LTS 2.8.0 which results in LTS 2.8.1
211 being released. Note that here we don’t allow the extra testing and debugging
212 time that we had between Nov 2022 and early Feb 2023. This is because there
213 isn’t as much to test and debug as an annual LTS release has. Also companies
214 might want to deploy critical patches soon.
215- Nov 2023: TF-A 2.10 is released. Not shown in the diagram, at the same time
216 LTS 2.10-rc is made. It’s tested by partners for a couple of months.
217- Feb 2024: LTS 2.10.1 is released in early Feb. Now there are two LTS
218 branches: 2.8.1 and 2.10.1.
219
220Note that TFTF will follow similar branching model as TF-A LTS, i.e. there will
221be TFTF LTS 2.8.0 in Feb 2023, 2.8.1 (if new TFTF tests need to be added for
222the security advisory) when there is TF-A LTS 2.8.1 and so on.
223
224Maintainership
225--------------
226
227**Guidelines & Responsibilities**
228
229#. Maintainers shall be impartial and strive to work for the benefit of
230 the community
231#. Objective and well-defined merge criteria to avoid confusion and discussions
232 at random points in time when there is a "candidate" patch
233#. The maintainers shall explain the lifecycle of a patch to the community,
234 with a detailed description of the maximum time spent in each step
235#. Automate, automate, automate
236#. Reviewers should not focus too much on "what" and instead focus on "how"
237#. Constantly refine the merge criteria to include more partner use cases
238#. Ensure that all candidate patches flow from the main branch to all LTS branches
Govindraj Raja8f88d032025-01-08 18:44:38 -0600239#. Maintainers collaborate in the following discord channel -
240 https://discord.com/channels/1106321706588577904/1162029539761852436
241#. Maintainers discuss and provide updates about upcoming LTS releases in the above
242 mentioned discord channel.
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600243
244**Options**
245
246These are some options in the order of preference.
247
248#. Current set of :ref:`lts maintainers` from tf.org(or hired contractor) take care of the LTS
249#. From the community, create a set of maintainers focused solely on the LTS branches
250
251A day in the life of a maintainer
252*********************************
253This section documents the daily tasks that a maintainer might perform to
254support the LTS program. It is expected that a maintainer follows clearly laid
255down steps and does not have to make policy level decisions for merge, testing,
256or candidate patch selection.
257
258#. Monitor the main branch to identify candidate patches for the LTS branches
Govindraj Raja8f88d032025-01-08 18:44:38 -0600259#. Monitor emails from LTS triage report to choose patches that should be
260 cherry-picked for LTS branches.
261#. Cherry-pick agreed patches to LTS branches co-ordinate review process and Monitor
262 CI results.
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600263#. Monitor the mailing list for any LTS related issues
264#. Propose or solicit patches to the main branch and tag them as candidates for LTS
265
266Execution Plan
267**************
268This section lists the steps needed to put the LTS system in place. However,
269to kick start LTS in Nov ‘22, only a few steps are needed. The rest can follow
270in the background.
271
272Initial release steps
273*********************
274
275The following steps are necessary to kickstart the project and potentially
276create the first LTS from the Nov’22 release.
277
278#. Create a TF-A LTS release-candidate branch and a TFTF LTS branch immediately
279 after the Nov’22 release
280#. Request all platform-owners to test and debug the RC branch
281#. Gather feedback from the test and debug cycle
282#. Mark the TF-A LTS branch ready by the end of January
283#. Announce the official LTS release availability on the mailing lists
284
285Long term release plan
286**********************
287Above will buy us time to then work on the rest of the execution plan which
288is given below.
289
290#. The review criteria for LTS patches must be the same as TF-A patches
291#. The maintainers shall publish the well-defined merge criteria to allow
292 the community to choose candidate patches
293#. The maintainers shall publish a well-defined test specification for any
294 patch entering the LTS branch
295
296 a. Tests required to pass in the CI/CD flow
297 b. Static analysis scans
298 c. Coverity scans
299
300#. The maintainers shall publish a mechanism to choose candidate patches for
301 the LTS branch
302#. The maintainers shall publish a mechanism to report bugs `[1]`_ seen with
303 an LTS branch
304#. The maintainers shall publish a versioning mechanism for the LTS branch
305
Govindraj Raja8f88d032025-01-08 18:44:38 -0600306 a. Bump minor version for any “logical” `[2]`_ fix(es) that gets merged
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600307
308#. The CI/CD infrastructure shall provide test support for all “live” LTS
309 branches at any given point in time
310#. The CI/CD infrastructure shall provide means to
311
312 a. notify all maintainers that a patch is ready for review
313 b. automatically cherry-pick a patch to a given LTS branch
314 c. get it through the CI/CD testing flow
Govindraj Raja8f88d032025-01-08 18:44:38 -0600315 d. gentle ping in LTS discord channel asking for reviews to ensure
316 cherry-picks are merged.
Govindraj Rajaae67fb42024-12-12 17:16:03 -0600317
318FAQ
319***
320
321In our discussions, in addition to the above points we also considered some
322questions. They have been discussed on the mailing list too.
323
324| Q. What happens when a bug fix applies just to a LTS branch and not to the
325 master branch?
326| A. This will be treated as a special case and the bug, and the fix will be
327 discussed
328
329| Q. When testing a backported patch, what if one of the partners needs more
330 time while the patch fix is time-critical and, hence slowing other
331 partners?
332| A. The maintainers will add more detail to the review and merge process to
333 handle this scenario.
334
335| Q. How do we handle the increasing version numbers for errata fixes?
336| A. Too many CPU errata workarounds resulting in too many LTS releases.
337 We propose bumping the version number for each logical fix as
338 described in the section “Long term release plan” above because
339 that will help accurately track what changes have been deployed in-field.
340
341| Q. What if LTS support duration needs to be extended to longer than 5 years?
342| A. Still under discussion.
343
344These are uncharted waters, and we will face some unseen problems. When they
345become real problems, then we will have concrete data and be better able to
346address them. This means that our LTS definition as presented in this document
347is not the final one. We will constantly be discussing it and deciding how to
348adapt it as we see practical problems.
349
350.. _[1]:
351
352[1] The plan is to create a system where reviewers can tag a patch on mainline which
353gets automatically rebased on LTS and pushed to Gerrit. On seeing this patch,
354the CI/CD starts tests and provides a score. In parallel, the system also sends
355an email to the maintainers announcing the arrival of a candidate patch for the
356LTS branch.
357
358.. _[2]:
359
360[2] Logical will be a patch or patches implementing a certain fix. For example, if a
361security mitigation is fixed with the help of three patches, then all of them are
362considered as one "logical" fix. The version is incremented only after all these
363patches are merged. with the maintainers. If agreed unanimously, the bug fix
364will be merged to the affected LTS branches after completing the review process.