1 .. SPDX-License-Identifier: BSD-3-Clause
2 Copyright 2021 The DPDK contributors
4 .. include:: <isonum.txt>
9 .. **Read this first.**
11 The text in the sections below explains how to update the release notes.
13 Use proper spelling, capitalization and punctuation in all sections.
15 Variable and config names should be quoted as fixed width text:
18 Build the docs and view the output file to ensure the changes are correct::
21 xdg-open build/doc/guides/html/rel_notes/release_21_11.html
27 .. This section should contain new features added in this release.
30 * **Add a title in the past tense with a full stop.**
32 Add a short 1-2 sentence description in the past tense.
33 The description should be enough to allow someone scanning
34 the release notes to understand the new feature.
36 If the feature adds a lot of sub-features you can use a bullet list
39 * Added feature foo to do something.
40 * Enhanced feature bar to do something else.
42 Refer to the previous release notes for examples.
44 Suggested order in release notes items:
45 * Core libs (EAL, mempool, ring, mbuf, buses)
46 * Device abstraction libs and PMDs (ordered alphabetically by vendor name)
48 - cryptodev (lib, PMDs)
49 - eventdev (lib, PMDs)
52 * Apps, Examples, Tools (if significant)
54 This section is a comment. Do not overwrite or remove it.
55 Also, make sure to start the actual text at the margin.
56 =======================================================
58 * **Enabled new devargs parser.**
60 * Enabled devargs syntax
61 ``bus=X,paramX=x/class=Y,paramY=y/driver=Z,paramZ=z``
62 * Added bus-level parsing of the devargs syntax.
63 * Kept compatibility with the legacy syntax as parsing fallback.
65 * **Updated EAL hugetlbfs mount handling for Linux.**
67 * Modified to allow ``--huge-dir`` option to specify a sub-directory
68 within a hugetlbfs mountpoint.
70 * **Added dmadev library.**
72 * Added a DMA device framework for management and provision of
73 hardware and software DMA devices.
74 * Added generic API which support a number of different DMA
76 * Added multi-process support.
78 * **Added IDXD dmadev driver implementation.**
80 The IDXD dmadev driver provide device drivers for the Intel DSA devices.
81 This device driver can be used through the generic dmadev API.
83 * **Added IOAT dmadev driver implementation.**
85 The Intel I/O Acceleration Technology (IOAT) dmadev driver provides a device
86 driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake,
87 Skylake and Broadwell. This device driver can be used through the generic dmadev API.
89 * **Added support to get all MAC addresses of a device.**
91 Added ``rte_eth_macaddrs_get`` to allow user to retrieve all Ethernet
92 addresses assigned to given ethernet port.
94 * **Added new RSS offload types for IPv4/L4 checksum in RSS flow.**
96 Added macros ETH_RSS_IPV4_CHKSUM and ETH_RSS_L4_CHKSUM, now IPv4 and
97 TCP/UDP/SCTP header checksum field can be used as input set for RSS.
99 * **Added L2TPv2 and PPP protocol support in flow API.**
101 Added flow pattern items and header formats of L2TPv2 and PPP protocol.
103 * **Added flow flex item.**
105 The configurable flow flex item provides the capability to introduce
106 an arbitrary user-specified network protocol header,
107 configure the hardware accordingly, and perform match on this header
108 with desired patterns and masks.
110 * **Added ethdev support to control delivery of Rx metadata from the HW to the PMD.**
112 A new API, ``rte_eth_rx_metadata_negotiate()``, was added.
113 The following parts of Rx metadata were defined:
115 * ``RTE_ETH_RX_METADATA_USER_FLAG``
116 * ``RTE_ETH_RX_METADATA_USER_MARK``
117 * ``RTE_ETH_RX_METADATA_TUNNEL_ID``
119 * **Added an API to get a proxy port to manage "transfer" flows.**
121 A new API, ``rte_flow_pick_transfer_proxy()``, was added.
123 * **Added ethdev shared Rx queue support.**
125 * Added new device capability flag and Rx domain field to switch info.
126 * Added share group and share queue ID to Rx queue configuration.
127 * Added testpmd support and dedicate forwarding engine.
129 * **Updated af_packet ethdev driver.**
131 * Default VLAN strip behavior was changed. VLAN tag won't be stripped
132 unless ``DEV_RX_OFFLOAD_VLAN_STRIP`` offload is enabled.
134 * **Added API to get device configuration in ethdev.**
136 Added an ethdev API which can help users get device configuration.
138 * **Updated AF_XDP PMD.**
140 * Disabled secondary process support.
142 * **Updated Amazon ENA PMD.**
144 Updated the Amazon ENA PMD. The new driver version (v2.5.0) introduced
145 bug fixes and improvements, including:
147 * Support for the tx_free_thresh and rx_free_thresh configuration parameters.
148 * NUMA aware allocations for the queue helper structures.
149 * Watchdog's feature which is checking for missing Tx completions.
151 * **Updated Broadcom bnxt PMD.**
153 * Added flow offload support for Thor.
154 * Implement support for tunnel offload.
155 * Updated HWRM API to version 1.10.2.44
157 * **Updated Cisco enic driver.**
159 * Added rte_flow support for matching GTP, GTP-C and GTP-U headers.
161 * **Updated Intel e1000 emulated driver.**
163 * Added Intel e1000 support on Windows.
165 * **Updated Intel iavf driver.**
167 * Added Intel iavf support on Windows.
168 * Added IPv4 and L4 (TCP/UDP/SCTP) checksum hash support in RSS flow.
169 * Added PPPoL2TPv2oUDP RSS hash based on inner IP address and TCP/UDP port.
170 * Added Intel iavf inline crypto support.
172 * **Updated Intel ice driver.**
174 * Added protocol agnostic flow offloading support in Flow Director.
175 * Added protocol agnostic flow offloading support in RSS hash.
176 * Added 1PPS out support by a devargs.
177 * Added IPv4 and L4 (TCP/UDP/SCTP) checksum hash support in RSS flow.
178 * Added DEV_RX_OFFLOAD_TIMESTAMP support.
179 * Added timesync API support under scalar path.
181 * **Updated Intel ixgbe driver.**
183 * Added Intel ixgbe support on Windows.
185 * **Updated Marvell cnxk ethdev driver.**
187 * Added rte_flow support for dual VLAN insert and strip actions.
188 * Added rte_tm support.
189 * Added support for Inline IPsec for CN9K event mode and CN10K
190 poll mode and event mode.
191 * Added support for ingress meter for CN10K platform.
193 * **Updated Mellanox mlx5 driver.**
195 Updated the Mellanox mlx5 driver with new features and improvements, including:
197 * Added implicit mempool registration to avoid data path hiccups (opt-out).
198 * Added NIC offloads for the PMD on Windows (TSO, VLAN strip, CRC keep).
199 * Added socket direct mode bonding support.
201 * **Updated Solarflare network PMD.**
203 Updated the Solarflare ``sfc_efx`` driver with changes including:
205 * Added port representors support on SN1000 SmartNICs
206 * Added flow API transfer proxy support
207 * Added SN1000 virtual functions (VF) support
208 * Added support for flow counters without service cores
210 * **Added power monitor API in vhost library.**
212 Added an API to support power monitor in vhost library.
214 * **Updated vhost PMD.**
216 Add power monitor support in vhost PMD.
218 * **Updated virtio PMD.**
220 * Initial support for RSS receive mode has been added to the Virtio PMD,
221 with the capability for the application to configure the hash key,
222 the RETA and the hash types.
223 Virtio hash reporting is yet to be added.
224 * Added power monitor support in virtio PMD.
226 * **Updated Wangxun ngbe driver.**
228 * Added offloads and packet type on RxTx.
229 * Added VLAN and MAC filters.
230 * Added device basic statistics and extended stats.
231 * Added multi-queue and RSS.
233 * Added flow control.
236 * **Added telemetry callbacks to cryptodev library.**
238 Added telemetry callback functions which allow a list of crypto devices,
239 stats for a crypto device, and other device information to be queried.
240 Also added callback to get cryptodev capabilities.
242 * **Updated Marvell cnxk crypto PMD.**
244 * Added AES-CBC SHA1-HMAC support in lookaside protocol (IPsec) for CN10K.
245 * Added Transport mode support in lookaside protocol (IPsec) for CN10K.
246 * Added UDP encapsulation support in lookaside protocol (IPsec) for CN10K.
247 * Added support for lookaside protocol (IPsec) offload for CN9K.
248 * Added support for ZUC algorithm with 256-bit key length for CN10K.
249 * Added support for CN98xx dual block.
250 * Added inner checksum support in lookaside protocol (IPsec) for CN10K.
251 * Added AES-CBC NULL auth support in lookaside protocol (IPsec) for CN10K.
252 * Added ESN and anti-replay support in lookaside protocol (IPsec) for CN9K.
254 * **Added support for event crypto adapter on Marvell CN10K and CN9K.**
256 * Added event crypto adapter OP_FORWARD mode support.
258 * **Updated Mellanox mlx5 crypto driver.**
260 * Added Windows support.
262 * **Updated NXP dpaa_sec crypto PMD.**
264 * Added DES-CBC, AES-XCBC-MAC, AES-CMAC and non-HMAC algo support.
265 * Added PDCP short MAC-I support.
266 * Added raw vector datapath API support.
268 * **Updated NXP dpaa2_sec crypto PMD.**
270 * Added PDCP short MAC-I support.
271 * Added raw vector datapath API support.
273 * **Added framework for consolidation of IPsec_MB dependent SW Crypto PMDs.**
275 * The IPsec_MB framework was added to share common code between Intel
276 SW Crypto PMDs that depend on the intel-ipsec-mb library.
277 * Multiprocess support was added for the consolidated PMDs,
278 which requires v1.1 of the intel-ipsec-mb library.
279 * The following PMDs were moved into a single source folder,
280 however their usage and EAL options remain unchanged.
286 * CHACHA20_POLY1305 - A new PMD added.
288 * **Updated the aesni_mb crypto PMD.**
290 * Added support for ZUC-EEA3-256 and ZUC-EIA3-256.
292 * **Added digest appended ops support for Snow3G PMD.**
294 * Added support for out-of-place auth-cipher operations that encrypt
295 the digest along with the rest of the raw data.
296 * Added support for partially encrypted digest when using auth-cipher
299 * **Updated the ACC100 bbdev PMD.**
301 Added support for more comprehensive CRC options.
303 * **Updated the turbo_sw bbdev PMD.**
305 Added support for more comprehensive CRC options.
307 * **Added NXP LA12xx baseband PMD.**
309 * Added a new baseband PMD driver for NXP LA12xx Software defined radio.
310 * See the :doc:`../bbdevs/la12xx` for more details.
312 * **Updated Mellanox compress driver.**
314 * Added devargs option to allow manual setting of Huffman block size.
316 * **Updated Mellanox regex driver.**
318 * Added support for new ROF file format.
320 * **Updated IPsec library.**
322 * Added support for more AEAD algorithms AES_CCM, CHACHA20_POLY1305
324 * Added support for NAT-T / UDP encapsulated ESP.
325 * Added support for SA telemetry.
326 * Added support for setting a non default starting ESN value.
327 * Added support for TSO in inline crypto mode.
329 * **Added optimized Toeplitz hash implementation.**
331 Added optimized Toeplitz hash implementation using Galois Fields New Instructions.
333 * **Added multi-process support for testpmd.**
335 Added command-line options to specify total number of processes and
336 current process ID. Each process owns subset of Rx and Tx queues.
338 * **Updated test-crypto-perf application with new cases.**
340 * Added support for asymmetric crypto throughput performance measurement.
341 Only modex is supported for now.
342 * Added support for lookaside IPsec protocol offload throughput measurement.
344 * **Added lookaside protocol (IPsec) tests in dpdk-test.**
346 * Added known vector tests (AES-GCM 128, 192, 256).
347 * Added tests to verify error reporting with ICV corruption.
348 * Added tests to verify IV generation.
349 * Added tests to verify UDP encapsulation.
350 * Added tests to verify UDP encapsulation ports.
351 * Added tests to validate packets soft expiry.
352 * Added tests to validate packets hard expiry.
353 * Added tests to verify tunnel header verification in IPsec inbound.
354 * Added tests to verify inner checksum.
355 * Added tests for CHACHA20_POLY1305 PMD, including a new testcase for SGL OOP.
357 * **Updated IPsec Security Gateway sample application with new features.**
359 * Added support for TSO (only for inline crypto TCP packets).
361 * **Revised packet capture framework.**
363 * New dpdk-dumpcap program that has most of the features
364 of the wireshark dumpcap utility including:
365 capture of multiple interfaces, filtering,
366 and stopping after number of bytes, packets.
367 * New library for writing pcapng packet capture files.
368 * Enhancements to the pdump library to support:
369 * Packet filter with BPF.
370 * Pcapng format with timestamps and meta-data.
371 * Fixes packet capture with stripped VLAN tags.
373 * **Added ASan support.**
376 <https://github.com/google/sanitizers/wiki/AddressSanitizer>`_ (ASan)
377 is a widely-used debugging tool to detect memory access errors.
378 It helps to detect issues like use-after-free, various kinds of buffer
379 overruns in C/C++ programs, and other similar errors, as well as
380 printing out detailed debug information whenever an error is detected.
386 .. This section should contain removed items in this release. Sample format:
388 * Add a short 1-2 sentence description of the removed item
391 This section is a comment. Do not overwrite or remove it.
392 Also, make sure to start the actual text at the margin.
393 =======================================================
395 * eal: Removed the deprecated function ``rte_get_master_lcore()``
396 and the iterator macro ``RTE_LCORE_FOREACH_SLAVE``.
398 * eal: The old api arguments that were deprecated for
399 blacklist/whitelist are removed. Users must use the new
400 block/allow list arguments.
402 * mbuf: Removed offload flag ``PKT_RX_EIP_CKSUM_BAD``.
403 ``PKT_RX_OUTER_IP_CKSUM_BAD`` should be used as a replacement.
405 * ethdev: Removed the port mirroring API. A more fine-grain flow API
406 action ``RTE_FLOW_ACTION_TYPE_SAMPLE`` should be used instead.
407 The structures ``rte_eth_mirror_conf`` and ``rte_eth_vlan_mirror`` and
408 the functions ``rte_eth_mirror_rule_set`` and
409 ``rte_eth_mirror_rule_reset`` along with the associated macros
410 ``ETH_MIRROR_*`` are removed.
412 * ethdev: Removed ``rte_eth_rx_descriptor_done`` API function and its
413 driver callback. It is replaced by the more complete function
414 ``rte_eth_rx_descriptor_status``.
416 * ethdev: Removed deprecated ``shared`` attribute of the
417 ``struct rte_flow_action_count``. Shared counters should be managed
418 using indirect actions API (``rte_flow_action_handle_create`` etc).
420 * i40e: Removed i40evf driver.
421 iavf already became the default VF driver for i40e devices,
422 so there is no need to maintain i40evf.
428 .. This section should contain API changes. Sample format:
430 * sample: Add a short 1-2 sentence description of the API change
431 which was announced in the previous releases and made in this release.
432 Start with a scope label like "ethdev:".
433 Use fixed width quotes for ``function_names`` or ``struct_names``.
436 This section is a comment. Do not overwrite or remove it.
437 Also, make sure to start the actual text at the margin.
438 =======================================================
440 * eal: The lcore state ``FINISHED`` is removed from
441 the ``enum rte_lcore_state_t``.
442 The lcore state ``WAIT`` is enough to represent the same state.
444 * eal: Made ``rte_intr_handle`` structure definition hidden.
446 * kvargs: The experimental function ``rte_kvargs_strcmp()`` has been
447 removed. Its usages have been replaced by a new function
448 ``rte_kvargs_get_with_value()``.
450 * cmdline: ``cmdline_stdin_exit()`` now frees the ``cmdline`` structure.
451 Calls to ``cmdline_free()`` after it need to be deleted from applications.
453 * cmdline: Made ``cmdline`` structure definition hidden on Linux and FreeBSD.
455 * cmdline: Made ``rdline`` structure definition hidden. Functions are added
456 to dynamically allocate and free it, and to access user data in callbacks.
458 * mempool: Added ``RTE_MEMPOOL_F_NON_IO`` flag to give a hint to DPDK components
459 that objects from this pool will not be used for device IO (e.g. DMA).
461 * mempool: The mempool flags ``MEMPOOL_F_*`` will be deprecated in the future.
462 Newly added flags with ``RTE_MEMPOOL_F_`` prefix should be used instead.
464 * mempool: Helper macro ``MEMPOOL_HEADER_SIZE()`` is deprecated.
465 The replacement macro ``RTE_MEMPOOL_HEADER_SIZE()`` is internal only.
467 * mempool: Macro to register mempool driver ``MEMPOOL_REGISTER_OPS()`` is
468 deprecated. Use replacement ``RTE_MEMPOOL_REGISTER_OPS()``.
470 * mempool: The mempool API macros ``MEMPOOL_PG_*`` are deprecated and
471 will be removed in DPDK 22.11.
473 * mbuf: The mbuf offload flags ``PKT_*`` are renamed as ``RTE_MBUF_F_*``. A
474 compatibility layer will be kept until DPDK 22.11.
475 * net: Renamed ``s_addr`` and ``d_addr`` fields of ``rte_ether_hdr`` structure
476 to ``src_addr`` and ``dst_addr``, respectively.
478 * net: Added ``version`` and ``ihl`` bit-fields to ``struct rte_ipv4_hdr``.
479 Existing ``version_ihl`` field is kept for backward compatibility.
481 * ethdev: Added items and actions ``PORT_REPRESENTOR``, ``REPRESENTED_PORT`` to
484 * ethdev: Deprecated items and actions ``PF``, ``VF``, ``PHY_PORT``, ``PORT_ID``.
485 Suggested items and actions ``PORT_REPRESENTOR``, ``REPRESENTED_PORT`` instead.
487 * ethdev: Deprecated the use of attributes ``ingress`` / ``egress`` combined
488 with ``transfer``. See items ``PORT_REPRESENTOR``, ``REPRESENTED_PORT``.
490 * ethdev: ``rte_flow_action_modify_data`` structure updated, immediate data
491 array is extended, data pointer field is explicitly added to union, the
492 action behavior is defined in more strict fashion and documentation updated.
493 The immediate value behavior has been changed, the entire immediate field
494 should be provided, and offset for immediate source bitfield is assigned
495 from destination one.
497 * vhost: ``rte_vdpa_register_device``, ``rte_vdpa_unregister_device``,
498 ``rte_vhost_host_notifier_ctrl`` and ``rte_vdpa_relay_vring_used`` vDPA
499 driver interface are marked as internal.
501 * cryptodev: The API rte_cryptodev_pmd_is_valid_dev is modified to
502 rte_cryptodev_is_valid_dev as it can be used by the application as
503 well as PMD to check whether the device is valid or not.
505 * cryptodev: The rte_cryptodev_pmd.* files are renamed as cryptodev_pmd.*
506 as it is for drivers only and should be private to DPDK, and not
507 installed for app use.
509 * cryptodev: A ``reserved`` byte from structure ``rte_crypto_op`` was
510 renamed to ``aux_flags`` to indicate warnings and other information from
511 the crypto/security operation. This field will be used to communicate
512 events such as soft expiry with IPsec in lookaside mode.
514 * cryptodev: The field ``dataunit_len`` of the ``struct rte_crypto_cipher_xform``
515 moved to the end of the structure and extended to ``uint32_t``.
517 * cryptodev: The structure ``rte_crypto_vec`` was updated to add ``tot_len``
518 field to support total buffer length to facilitate protocol offload case.
520 * cryptodev: The structure ``rte_crypto_sym_vec`` was updated to add
521 ``dest_sgl`` to support out of place processing.
523 * bbdev: Added device info related to data byte endianness processing.
525 * eventdev: Moved memory used by timer adapters to hugepage. This will prevent
526 TLB misses if any and aligns to memory structure of other subsystems.
528 * fib: Added the ``rib_ext_sz`` field to ``rte_fib_conf`` and ``rte_fib6_conf``
529 so that user can specify the size of the RIB extension inside the FIB.
535 .. This section should contain ABI changes. Sample format:
537 * sample: Add a short 1-2 sentence description of the ABI change
538 which was announced in the previous releases and made in this release.
539 Start with a scope label like "ethdev:".
540 Use fixed width quotes for ``function_names`` or ``struct_names``.
543 This section is a comment. Do not overwrite or remove it.
544 Also, make sure to start the actual text at the margin.
545 =======================================================
547 * ethdev: All enums & macros updated to have ``RTE_ETH`` prefix and structures
548 updated to have ``rte_eth`` prefix. DPDK components updated to use new names.
550 * ethdev: Input parameters for ``eth_rx_queue_count_t`` was changed.
551 Instead of pointer to ``rte_eth_dev`` and queue index, now it accepts pointer
552 to internal queue data as input parameter. While this change is transparent
553 to user, it still counts as an ABI change, as ``eth_rx_queue_count_t``
554 is used by public inline function ``rte_eth_rx_queue_count``.
556 * ethdev: Made ``rte_eth_dev``, ``rte_eth_dev_data``, ``rte_eth_rxtx_callback``
557 private data structures. ``rte_eth_devices[]`` can't be accessed directly
558 by user any more. While it is an ABI breakage, this change is intended
559 to be transparent for both users (no changes in user app is required) and
560 PMD developers (no changes in PMD is required).
562 * vhost: rename ``struct vhost_device_ops`` to ``struct rte_vhost_device_ops``.
564 * cryptodev: Made ``rte_cryptodev``, ``rte_cryptodev_data`` private
565 structures internal to DPDK. ``rte_cryptodevs`` can't be accessed directly
566 by user any more. While it is an ABI breakage, this change is intended
567 to be transparent for both users (no changes in user app is required) and
568 PMD developers (no changes in PMD is required).
570 * security: ``rte_security_set_pkt_metadata`` and ``rte_security_get_userdata``
571 routines used by inline outbound and inline inbound security processing were
572 made inline and enhanced to do simple 64-bit set/get for PMDs that do not
573 have much processing in PMD specific callbacks but just 64-bit set/get.
574 This avoids a per packet function pointer jump overhead for such PMDs.
576 * security: A new option ``iv_gen_disable`` was added in structure
577 ``rte_security_ipsec_sa_options`` to disable IV generation inside PMD,
578 so that application can provide its own IV and test known test vectors.
580 * security: A new option ``tunnel_hdr_verify`` was added in structure
581 ``rte_security_ipsec_sa_options`` to indicate whether outer header
582 verification need to be done as part of inbound IPsec processing.
584 * security: A new option ``udp_ports_verify`` was added in structure
585 ``rte_security_ipsec_sa_options`` to indicate whether UDP ports
586 verification need to be done as part of inbound IPsec processing.
588 * security: A new structure ``rte_security_ipsec_lifetime`` was added to
589 replace ``esn_soft_limit`` in IPsec configuration structure
590 ``rte_security_ipsec_xform`` to allow applications to configure SA soft
591 and hard expiry limits. Limits can be either in number of packets or bytes.
593 * security: The new options ``ip_csum_enable`` and ``l4_csum_enable`` were added
594 in structure ``rte_security_ipsec_sa_options`` to indicate whether inner
595 packet IPv4 header checksum and L4 checksum need to be offloaded to
598 * security: A new structure ``esn`` was added in structure
599 ``rte_security_ipsec_xform`` to set an initial ESN value. This permits
600 application to start from an arbitrary ESN value for debug and SA lifetime
601 enforcement purposes.
603 * security: A new structure ``udp`` was added in structure
604 ``rte_security_ipsec_xform`` to allow setting the source and destination ports
605 for UDP encapsulated IPsec traffic.
607 * bbdev: Added capability related to more comprehensive CRC options,
608 shifting values of the ``enum rte_bbdev_op_ldpcdec_flag_bitmasks``.
610 * eventdev: New variables ``rx_event_buf_count`` and ``rx_event_buf_size``
611 were added in structure ``rte_event_eth_rx_adapter_stats`` to get additional
614 * eventdev: A new structure ``rte_event_fp_ops`` has been added which is now used
615 by the fastpath inline functions. The structures ``rte_eventdev``,
616 ``rte_eventdev_data`` have been made internal. ``rte_eventdevs[]`` can't be
617 accessed directly by user any more. This change is transparent to both
618 applications and PMDs.
620 * eventdev: Re-arranged fields in ``rte_event_timer`` to remove holes.
626 .. This section should contain new known issues in this release. Sample format:
628 * **Add title in present tense with full stop.**
630 Add a short 1-2 sentence description of the known issue
631 in the present tense. Add information on any known workarounds.
633 This section is a comment. Do not overwrite or remove it.
634 Also, make sure to start the actual text at the margin.
635 =======================================================
641 .. This section should contain a list of platforms that were tested
646 * <vendor> platform with <vendor> <type of devices> combinations
651 * Other relevant details...
653 This section is a comment. Do not overwrite or remove it.
654 Also, make sure to start the actual text at the margin.
655 =======================================================